Thanks for your paitence in guoding me ,I appreciate it

Shivam

One point to be noted that we are not looking for “runtime” manipulation of metadata for target schema or the process flow.

Based on customer specific requirements for target schema variation (elimination of columns or entities), we should be able to generate from scratch or tweak master metadata for target schema map or process flow to generate desired output. This is still considered as “design” time, just that on-site person is not using GUI tools to design the process flow rather our infrastructure will generate it.

Based on your evaluation, does Talend and Pentaho support this?

I will need to answer this from Imformatica perspective.

_______________________________________________________________________________________________________________________________________________________

Our Analysis:——————

steps:—

1. Create table control_load
2. Add 3 columns like tab_name,column_name,flag
3.Develop map to generate the parameter file where flag = ‘Y’ with parameter
as INSERT statement( for those columns or table you dont want make the flag as ‘N’)
4. Develop the actual map, after source add SQL transformation and call the parameter
for Inserting to target tableflow make dymamic for SQL transformation
5.In work

two mappings one workflow, call sessions in sequential mode
it covers both scenarios
based on flag you can do that
it all depends on how you’ll set the flag

Kirtan, make changes to the steps if i m wrong
Shivam

I am running a workflow, in this workgflow there are 2 session. one session is for loading data from file into staging table and 2nd session for loading data from staging into fact table.
first session is running fine , but when i run the 2nd session which has 65k records to load, it keep running for 1-1.5 hrs withour any error message, only message i found in logs is “CONNECTING to DATABASE” i checked every level in db i.e. checked the temp tablespace it has enough space.. and there’s no lock on the target table.
Need your support on this issue.

thanks in advance,
Nitin

in your mapping create a lookup to the table that you are going to load with the contents of this file.

in a filter transformation, discard the rows that are already in the DB/table.

AFTER that rank rest of the rows. let all the rows with rank < =10 pass through. and discard rest. OR discard where rank > 10.

————-
one of the biggest drawback of informatica in my opinion is that it does not and can not allow looping construct…

now the solution above would work fine if you are dealing with 100 rows (as you mentioned). however , if you are dealing with 200 million rows it can be a nightmare.

i would rather do something like the following if my source file was 200m and if i had a choice.

say your source file looks like:

ID NAME
1 A
2 B
3 C
4 D
5 E
…
…
26 Z

I would load it into a database table [ in staging area/schema/space sort of] called Table T.

So table T will look just like the file.

now, from table T to your main table where you really want to load this data, you can either loop through the data or run any type of SQL on it by providing range etc… your options are limitless from this point onwards…and the time/money/resource saved by not doing it using pure informatica code can be used for something else.

in ETL, remember this…
if you can do something using SQL and nothing else, use only SQL and nothing else
if you cannot do it in SQL , use PL/SQL or T-SQL or similar.
If you cannot do it in PL/SQL or T-SQL or similar, use Informatica
If you cannot do it in Informatica, use .NET/Java/Perl etc
If you cannot do it in .NET/Java/Perl etc, stop coding . go back to your analysis and find the biggest mistake/misunderstanding of your life.