Handling Large Payload Files

Problem:
Want to create a BPEL process to read, transform and translate a large payload file (e.g. size 1GB)?

Thoughts:
Reading and traversing large payload file was always a problem in SOA 10g. Supposedly Maximum limit for such operations are for less than 7MB files in 10g.
Solution to such problems are answered in 11g where you can read and transform huge payload files.

Solution:
Suppose you have a CSV file incoming from source directory which you want to transform to a Fixed length file. Keep the source and the destination schema (XSD) ready. Also prepare the Transformation file (XSL) and keep it handy.
For accomplishing this, you will only be using one File Adapter which will take care of your read, transform and write - so all the three operations takes place under single I/O interaction.

Follow the steps below for making it possible:
1) Drag and drop a file adapter in the external references swim lane. (For e.g. Give name as FileMove.)
2) Select Synchronous File Read
3) Give any dummy value for File physical path and File Name. It will be changed manually later.
4) Select Native format translation is not required (Opaque Schema)
5) Outbound File Adapter is now configured, click on Finish
6) Open the relevant .jca file (in this case FileMove_file.jca)
7) Ensure that className should be
"oracle.tip.adapter.file.outbound.FileIoInteractionSpec"
8) Add extra parameters as shown in below file:
<adapter-config name="FileMove" adapter="File Adapter"
xmlns="http://platform.integration.oracle/blocks/adapter/fw/metadata">
<connection-factory location="eis/FileAdapter" adapterRef=""/>
<endpoint-interaction portType="FileMove_ptt" operation="FileMove">
<interaction-spec
className="oracle.tip.adapter.file.outbound.FileIoInteractionSpec">
<property name="SourcePhysicalDirectory" value="testDir1"/>
<property name="SourceFileName" value="test1"/>
<property name="SourceSchema" value="xsd/source-csv.xsd"/>
<property name="SourceSchemaRoot value="Root-Element"/>
<property name="SourceType" value="native"/>
<property name="TargetPhysicalDirectory" value="testDir2"/>
<property name="TargetFileName" value="test2"/>
<property name="TargetSchema" value="xsd/destination-fixedLength.xsd"/>
<property name="TargetSchemaRoot value="Root-Element"/>
<property name="TargetType" value="native"/>
<property name="Xsl value="xsl/SourceToDestination.xsl"/>
<property name="Type" value="MOVE"/>
</interaction-spec>
</endpoint-interaction>
</adapter-config>
9) Save the file and deploy the process.

Note:
This will works only if all the records in the data file are of the same type.

File Adapter: File Age parameter usage

Problem:
What is the function of using File Age parameter in File Adapter?

Solution:
When user sets ‘File Age’ parameter in File Adapter wizard, then the BPEL will wait for ‘File Age’ time, before it first polls for a file.
This option is especially useful, when the source file is large and takes some time for an external application to completely write finishing the source file. If this is not done then there is a chance that incomplete source file may be picked up for processing by BPEL.
The actual value to be set for ‘File Age’ is decided upon the file size and the time it roughly takes to completely copy in to source folder for BPEL to pick up.

ESB:rejectedMessageHandlers

Problem:
Files which do not adhere to input file schemas should be rejected or moved to specified folder other than archive folder in ESB.

Solution:
For this, first create an ESB project. Add a Read file Adapter to it. Now to set the rejectedMessageHandlers property, we need to open the <Read adapter name>.esbsvc file. This file should be present under ESB project folder.

Add the following code just after the closing tag of invocation i.e. </invocation> and just before the </service> tag.

<endpointProperties>
<property name="rejectedMessageHandlers" value="file://C:\SOADir\reject"/>
</endpointProperties>

Here,
C:\SOADir\reject is my directory where I want my rejected files to come.

Note: If at anytime you try to refresh anything in the project, this property gets omitted automatically. So be sure to add it again in .esbsvc file if you are refreshing anything in the project.

BPEL:rejectedMessageHandlers

Problem:
Files which do not adhere to the input file schemas should be rejected or moved to specified folder other than archive folder.

Solution:
For this we need to make an empty project which will be initiated by a file. We need to use a Read File Adapter. After the Adapter is configured and connected to a Receive activity, a bpel.xml file will be generated.
A property called ‘rejectedMessageHandlers’ is used and set in the bpel.xml file under <activationAgent> element.

Sample code for this should look like:
<activationAgents>
<activationAgent className="oracle.tip.adapter.fw.agent.jca.JCAActivationAgent" partnerLink="Read_File">
<property name="portType">Read_ptt
<property name="rejectedMessageHandlers">file://C:\SOADir\reject
</activationAgent>
</activationAgents>

Here,
C:\SOADir\reject is my directory where I want my rejected files to come.

Note: If at anytime you try to refresh anything in the BPEL project, this property gets omitted automatically. So be sure to add it again in bpel.xml file if you are refreshing any compomnent in the project.

Header Information in ESB

Problem:
To retrieve Header Information in ESB

Solution:

For FTP Adapter:
1) Create a variable in XSL mapper
2) Use the code in XPath:
ehdr:getRequestHeader("/fhdr:InboundFTPHeaderType/fhdr:fileName","fhdr=http://xmlns.oracle.com/pcbpel/adapter/ftp/;")

For File Adapter:
1) Create a variable in XSL mapper
2) Use the code in XPath:
ehdr:getRequestHeader("/fhdr:InboundFileHeaderType/fhdr:fileName","fhdr=http://xmlns.oracle.com/pcbpel/adapter/file/;")

More options in Native Format Builder

INTRODUCTION:
This wizard helps you in creating XSDs for your sample files. This is used to create a native schema file from formats such as:- comma-separated value (CSV), fixed-length file, data type description (DTD), and Cobol Copybook.

SCENARIO:
Suppose you need to create a schema file for a pipe-delimeter or $ delimeter sample file. If you closely look into the options provided under Step 5 of 7 (If you select Delimited Radio button in Step1 of 7) in Native Format Builder, you will see only 5 options available i.e. Single Space, Comma, Semicolon, White space (any number of tab, space) and Tab.
Then how to create a pipe-delimeter or $ delimeter schema file??

SOLUTION:
Goto options provided under Step 5 of 7 (If you select Delimited Radio button in Step1 of 7) in Native Format Builder.
In the Fields ->Delimited by drop down option, double click. A pointer will appear. Now instead of using drop down option manually type the |(pipe).

Then finish off the wizard. Your work is done..So simple.
The pipe delimited file schema file is ready to use.

Keep learning and stay connected to SOA.

WISHING ALL THE BLOG READERS A VERY HAPPY NEW YEAR 2009.

File Adapter : Hidden Treasure-2

Getting File Size

I am assuming that people reading this blog know how to get the File name and the input directory of the file. In this blog entry I will demonstrate an another aspect of File Adapter i.e. getting the input file size.
Note:- If you don't know about getting file name & input directory, refer to my blog entry dated 14th June 2008.

Introduction:
If you see the fileAdapterInboundHeader.wsdl file in Jdeveloper, you will notice that this wsdl is containing only two elements i.e. fileName and directory. Now If you goto the BPEL Console and look closely at the variable part (which contains file name and directory information), you will notice that at runtime Assign activity is displaying 5 elements. So I felt that somehow I can get the file size too. Its pretty simple.

Solution:
To acheive it, just you have to modify the fileAdapterInboundHeader.wsdl file. Note that the fileAdapterInboundHeader.wsdl is a read-only file so to modify it, you have to remove the read-only permission. After removing the permission you can edit the code.

After change, the code will look like this:


Note the highlighted element "size" below the directory element. After doing the changes, apply the read-only permission back to the file(not necessary).

Now you will be able use the size attribute in your assign activity.

Pretty cool stuff...right..

Have a Happy Happy learning...

File Adapter : Hidden Treasure-1

Working on Logical Name Option of File adapter

This blog entry is for those who have worked with File Adapter a bit. In this blog I will mainly focus on showing you all the working of Logical Name option under Read operation for ESB.
We all are aware that while dealing with File Adapter Read operation, we can provide the input directory path in the form of either Physical Path or Logical Name.

ESB: The main problem in case of ESB is that we dont have files like bpel.xml to give details for the working of Logical Name.
Below I will show you how to deal with Logical Name with ESB.

1) Create an ESB project. Create a read file adapter and give Logical Name detail as shown below (Note: You can give Logical Name for Archive files also. Give different logical Names for incoming files and Archive. You cannot use same name)



Complete the read Adapter wizard.

2) Now automatically one routing service will be created by the wizard. Double click to open it.

3)The inbound file adapter window similar to the one shown below will be displayed with name like <SystemName_ServiceName>.esbsvc



4)When you click the plus sign (+) in Endpoint Properties section, the Endpoint Property Chooser dialog box will be displayed.



5) Select the InputDirectory property and click OK. Now you can specify the physical directory name in the Value field as shown below



6) Save it and proceed with your project.

Note: If you want to use Logical Name for Write operation of File Adapter same steps should be followed. Try it...

BPEL: If you are working on File Adapter Read operation and choosing the Logical Name option for Input Directory, you need to manipulate the bpel.xml file and provide the directory path there.

Thats it...You have learnt to use Logical Name option under File Adapter Read and write operation of ESB..

Happy Learning...

High Availability for Files

We can use the file adapter's high availability feature provided under SOA Suite 10.1.3.3. The following steps shown below will guide you to configure the file adapter for this feature:

1. Create a share folder on highly available file system.
Note:- This folder should have the write permission and should be accessible from all the systems that are running the file adapter.

2. Goto the following location:
SOA_HOME\bpel\system\service\config directory.

3. Open the file pc.properties.

4. Set oracle.tip.adapter.file.controldirpath to the shared folder name.
For example:
oracle.tip.adapter.file.controldirpath := Z:\\mysharedrive
or
oracle.tip.adapter.file.controldirpath := \\Myshare\Mydir

4. Restart the SOA Suite or Oracle BPEL Process Manager server.

Happy learning...

BPEL: Getting File Name and directory name from File Adapter

Problem:
How to know the file name and the directory name at run time that we selected during configuring File adapter in our BPEL Process?

Solution:
I will give the answer for this in step wise manner.
1) Create an empty BPEL process.
2) Create a Read File Adapters. This will create fileAdapterInboundHeader.wsdl and .wsdl file.
3) Drag receive activity to the swim lane and link it to the read operation and create variable(choose in receive activity). Check the create instance check box.
4) Create a variable of type InboundHeader_msg, say Varaiable_1 ( this message is chosen from the fileAdapterInboundHeader.wsdl).
5) Create two more variables of simple type (String), say Test1 and Test2.
5) Drag the assign activity. In the assign activity create a copy operation you can see the Variable_1 having two elements i.e. FileName and DirectoryName. You can point these two elements to the variable Test1 and Test2 respectively.
6) Now double click on receive activity go to the "Adapter" tab and choose the above Variable_1 as the Input Header Variable.

Similarly you can do it for File Write Adapter. In this case the fileAdapterOutboundHeader.wsdl contains only one element i.e. file name.

And you are done....Now you can use that value for further operations, depending upon your needs.

Have a happy learning.....

BPEL: File Adapter - Write Opeartion

Problem:Can we use File Adapter to write in a file in append mode. i.e. a BPEL process opening a file appending some text and closing the file each time it called.


Solution:
To write a file in append mode, you need to add one line in wsdl file.

Make an entry with Append="true"
inside tag found in wsdl file, created after Write file adapter.

PhysicalDirectory="C:\TMP"
InteractionSpec="oracle.tip.adapter.file.outbound.FileInteractionSpec"
FileNamingConvention="write.txt"
OpaqueSchema="true"
Append="true">

New to SOA?? : Useful Links

Some of the interesting links on SOA which is taken from Marc Kelderman blog are mentioned below. These links will be helpful for those who are planning to start and for those also who are still learning.
To start with Oracle SOA here are some of useful links:

Webservice standards :-
BPEL
WS-Addressing
WSDL
SOAP
XPath
XQuery
XSLT
XPath
WSIF

Oracle SOA Suite software :-
The Oracle SOA Suite
Oracle JDeveloper

Oracle SOA Suite general :-
Oracle Fusion Middleware (functional)
Oracle Fusion Middleware (technical)
Oracle SOA Suite
BPEL
ESB
Business Rules
Business Activity Monitor
Business Process Analysis

Installation :-
Installation Guide
ESB Performance patch
SOA/BPEL patches
General (Metalink -> Patches -> Advanced -> Product (SOA)BPEL)

Documentation :-
The Administration Guide
The BPEL Developers Guide
The ESB Developers Guide
The Adapters Guide
The Workflow Guide
Business Rules User Guide
Business Rules Reference
General oracle documentation
BPEL Correlation

Tutorials :-
SOA Quick start guide
SOA Tutorial (ESB/BPEL/Rules/Workflow)

Performance :-
BPEL Tuning
Tuning in a nutshell
SOA Monitor performance

API :-
BPEL Process Manager
Worflow API
Business Rules

Oracle Forum :-
Oracle SOA
Oracle BPEL

Oracle Blogs :-
My Own blog
Matt Wright'Blog
Antony Reynolds Blog
http://clemensblog.blogspot.com/
AMIS SOA blog
IT-Eye SOA

Oracle General :-
Oracle Technical Network (OTN)
Oracle Metalink (Support)

BPEL : How to read a file in middle of a process?

Question: How to read a file in middle of a BPEL process?

Answer: Following are the ways to read a file in mid of a BPEL process:

1. Using File Adapter: Avoid using the file adapter to read a file. The read file adapter is used to trigger the BPEL process in most cases. So in this scenario, the file adapter is not a good choice. Also it needs to use correlation sets to read files in the middle of a process.

2. Using Embedded Java code: It can solve the problem but it is not recommended and also it is hard to maintain.

3. Using built-in XPath function: The good way to read a file in the middle of a BPEL process is to use the built-in XPath function --> ora:readfile.

Example:

1. Drag an Assign activity in the swim lane at proper place

2. Create an Assign rule using the ora:readfile("file location", "xml schema path") XPath function

3. Assign the result to a variable.