Petals-SE-Talend

Version 10 by Vincent Zurczak
on Mar 05, 2010 10:57.

<< Changes from 9 to 10

compared with

Version 11 by Vincent Zurczak
on Mar 05, 2010 11:02.

Changes from 11 to 12 >>

Key

This line was removed.

This word was removed. This word was added.

This line was added.

Changes (1)

View Page History

...

{section}
{column}

h1. Features

This component allows to expose Talend jobs as services into Petals and to execute them inside the bus.
It only acts as service provider, not as a service consumer.
This component supports jobs created with both Talend Open Studio and Talend Integration Suite.

It provides several mechanisms to pass information and data to a job.
Il also supports several mechanisms to retrieve information and data from a job.

Before going further, it must be clear that no configuration for this component is intended to be created by hand.Neither by the Petals Studio.
In fact, only Talend Open Studio and Talend Integration Suite have the ability to generate a correct configuration for this component.
{column}
{column:width=300px}
{panel}{toc}{panel}
{column}{section}

h1. Message Processing

This section deals with the way messages (or requests) are processed by the Petals-SE-Talend component.
As a user, it is important to understand the logic of the component to use it efficiently.

There are five steps in the processing of a request.

h2. Validating the request

When a request is received and started to be processed in the Petals-SE-Talend component, it is validated before being really processed.
Here are the different steps involved in this validation process.

The first step is the WSDL-based validation of the request's XML payload.
If the *validate-exchange-by-wsdl* parameter is set to *true*, either in the component or in the service-unit, then the XML payload is validated against the WSDL of the service-unit.
If the validation fails, a fault is raised. Otherwise, the validation goes on.

{warning}

Be careful, WSDL-based validation does not work when the input message contains attachments.
The Talend export for Petals does prevent that from happening.
Just remember it if you modify the jbi.xml by hand.

{warning}

\\
The WSDL-based validation checks three elements:
# The called operation is defined in the service's WSDL.
# The called operation is associated with the called Message Exchange Pattern (MEP).
# The XML payload is validated against the WSDL's XML schemas.

{warning}
Be careful, the current implementation of this feature makes disk access, thus reducing the performances.
{warning}

Besides, it must be noticed that this component only supports *InOut* pattern.
No other pattern is or will be supported, unless the component undergoes a severe refactoring. It is up to the client to manage the call correctly.

\\

The second and last step in the validation is a check about the singleton property of a job.
If a job is singleton, it means that only one instance of this job can be executed at once.

...

{info}

One typical example of a singleton job is a job which moves data from one database to another one.
It would make no sense for two instances of this job to run at the same time, especially if they work on the same databases.
{info}

If the job is singleton and already running, then a fault is raised.
Otherwise, a new job instance is created. If the job is singleton, then the running state of this job is set to true and locked until it is this state is released (i.e. the job is executed).

\\
{note}

The job creation strategy is a lazy strategy. A job instance is created on every received and validated message.
The consequence for singleton jobs is that all the messages sent to a singleton job while it is running will be rejected.
{note}

Once accepted, the request can now be parsed to prepare the job's input.

h2. Preparing the job's input

Once the request has been accepted, it is parsed to get the different possible parameters for the job.
The message input contains up to 4 parts, that are described in the serivce's WSDL.
# The first parameters are the context parameters, child elements of the *contexts* element from the input message. These parameters will be passed to the job in its main method.
# Then, the data flow to be passed to a tPetalsIOnput instance is retrieved from the request. This data flow may not be present.
# The third kind of parameters is the input attachments.
# Eventually, the component processes the native options to be passed to the job.

{note}
If a job does not support to be passed data flow (for a tPetalsInput), an entry is logged, but no fault is raised. The execution goes on normally.
{note}

\\
Input attachments must respect some constraints:

* Each input attachments is serialized as a temporary file.
* Its location will be passed to the job through a context variable. This is why attachments are associated with context variables.
* Be careful, attachments are expected to be passed in MTOM mode. That is to say the attachment element has a grand-child element "xop:include" whose href attribute references an attachment.
* Besides, the name of the attachment element is the name of the context variable that will be associated with the temporary file location.

{info}
As a user, you do not have to worry about this appearing complexity.
The configuration and the WSDL creation are made by the tools, during the export.
And hopefully, clients to call such a service can be generated automatically from the WSDL.{info}

\\
As you can see, from one JBI message (an XML payload and attachments), the Petals-SE-Talend component gets at most 4 kinds of parameter to pass to the jobs.
Three of them are merged together, since they are passed as contexts to the job. The remaining one concerns the tPetalsInput data.
Notice that the input message may not define any of these parameters. In this case, the component will pass nothing to the job.

{note}
In fact, the WSDL content and the expected parameters depend on the job's content and on the defined options during the export operation.
{note}

h2. Executing the job

At this point, the Petals-SE-Talend has built the job instance and prepared the parameters.
If the job contains a tPetalsInput component, the data for this component is passed to the job.
The Talend contexts and options are then passed to the job, right before its execution is launched.

h2. Getting the job's output

The native job's output is an array of array of String, that is to say: {code:lang=java}String[][]{code}
This result may contain only an integer, in which case {code:lang=java}String[ 0 ][ 0 ]{code} is an integer and indicates the result of the job execution. Or, this array contains raw data (which is the case if the job contains a tBufferOutput).
This can be a solution to determine whether the job execution succedded or not. The Petals-SE-Talend does not do it. It is the responsibility of the client to make this check (since in fact, it depends on the job itself).

If the job contained a tPetalsOutput, then the output data flow is retreived from the job.

{note}
If a job does not support to be asked data flow (for a tPetalsOutput), an entry is logged, but no fault is raised. The execution goes on normally.
{note}

\\
Eventually, if it was specified during the job export that output attachments are to be expected after the job was executed, then they are taken back from the job.
These attachments must be passed from the job to the component through files. These files are loaded by the component in memory and then, deleted from the disk.
The deletion of these files is not an option. Letting them on the disk could represent important risks. Indeed, a malicious client could override the context on each call, thus creating an infinite number of files on the disk. Unfinite until the disk crashes, obviously.

Like input attachments, output attachments are returned in MTOM mode.
{note}

If a component expects output attachments to be returned by the job, and that this job does not support it, then a fault is thrown.
This can typically happen if you created your job with Talend Open Studio and exported a context as an "OUT-Attachment".
{note}

h2. Building the response

Now that everything has been gathered from the job, the response can be built and returned.
Hence, the response can count up to 3 parts:
# The job's result (remember, the array of array of String). This part is always returned.
# The output data beans, if the job contained a tPetalsOutput.
# The output attachments.

Like the input message, the structure of the output message is determined by the job content and the options which were checked during the export of the job for Petals.

h1. Talend Open Studio vs. Talend Integration Suite

Two products can be used to create Talend jobs and export them into Petals: Talend Open Studio, which is an open source product, and Talend Integration Suite, which is the upgraded version of Talend Open Studio (but not free).
There are slight differences between what can be made in these two products.

h2. Talend Open Studio

Talend Open Studio does not support any specific component for Petals.
It supports the following features for the Petals-SE-Component:
* Expose a context as a parameter into the service's WSDL.
* Pass attachment files to a job.
* Pass native parameters and options to a job.
* Get the job's execution result.

Other features are only supported by Talend Integration Suite.
However, it is possible to use alternative ways, depending on the desired feature.

h2. Talend Integration Suite

In Talend Integration Suite, all the features of Talend Open Studio are supported.
It also embeds additional capabilities to work with Petals ESB.

* You can use tPetalsInput and tPetalsOutput components in your jobs.
** These components respectively allow you to pass data from Petals to the job and from the job to Petals.
** These two components do not have any specific parameter. They just need their schema to be defined.
* You can simulate the Petals behavior for these two components, meaning you can execute your job in Talend Integration Suite if it contains a tPetalsInput or a tPetalsOutput component.
** Thus, for a tPetalsInput, you can provide what would be the Petals input from a file that will be loaded before the simulation.
** For the tPetalsOutput, the returned document will be printed into the console.
* Eventually, Talend Integration Suite allows to attach files in the responses.

h2. The Talend export for Petals

The export is the same in both Talend Open Studio and Talend Integration Suite.
The generated artifacts are the same in both of them. And the content of the jbi.xml is the same in both of them.
The only differences are the job's content (which Talend components have been added into the job) and the generated WSDL (which depends on the job content and the export options).

The service-unit structure and the jbi.xml are desribed farther.
This section only introduces the export options, their impact on the result.
The generated WSDL is not discussed in this documentation.

Here is a snapshot version of the Talend export for Petals.

!SimpleJobExport.jpg!

The target file is the location of the service-assembly to generate.
The job version meaning is explicit.

The export options are the following:

|| Export Option || Description ||
| Singleton job | True to make the job singleton. A singleton job can have only one instance running at once on a given Petals-SE-Talend component. |
| Generate the end-point | True to let Petals generate the end-point at deployment time. If false, the end-point name is the job name with the suffix "Endpoint". |
| Validate Petals messages | True to validate all the messages / requests against the WSDL. \\
Be careful, enabling this option reduces the performances (disk access). \\ |
| User routines | Embed the user routines in the service-unit. |
| Source files | True to embed the source files in the generated service-unit. |
| Jobs contexts | Select the context that will be used by default by the job. |

\\
Eventually, there is the edition link to specify how contexts should be exposed in the generated WSDL.

!SimpleJobExportContext1.jpg!

When this link is clicked, a dialog shows up.
It lists all the job contexts, with their name, the type they will be associated with if exported, and the way they are exported.
By default, no context is exported. Said differently, the export mode for all the contexts is *Not exported*.

\\
Here is a small description of the export modes.

|| Export Mode || Description ||
| Not exported | The context is not exported (not visible as a parameter). But the context can still be overridden using the native parameters (options) of the job. |
| Parameter | The context is exported as a parameter in the contexts. |
| In-Attachment | The context will be passed the location of a temporary file whose content was attached in the input message. |
| Out-Attachment | The context will be read after the job was executed.
* This context must point to a file.
* The file content will be read by the Petals-SE-Talend component and put as an attachment into the response. 
* The context name will be used as the attachment name.
* The file will be deleted by the component right after its content was loaded. |
| Parameter and Out-Attachment | A mix between the Parameter and the Out-Attachement modes.
* The context is exposed as a parameter.
* It will also be read after the job execution.
* The file will be deleted anyway.
* The advantage of this export mode is to define the output file destination dynamically. |

h1. Component Configuration

The component can be configured through its JBI descriptor file, as shown below.
{code:lang=xml}<?xml version="1.0" encoding="UTF-8"?>
<jbi:jbi
   version="1.0"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xmlns:petalsCDK="http://petals.ow2.org/components/extensions/version-5"
   xmlns:jbi="http://java.sun.com/xml/ns/jbi"
   xmlns:talend="http://petals.ow2.org/components/talend/version-1">

   <jbi:component type="service-engine">

       <jbi:identification>
           <jbi:name>petals-se-talend</jbi:name>
           <jbi:description>A service engine to expose and run Talend jobs as services in Petals</jbi:description>
       </jbi:identification>

       <jbi:component-class-name>org.ow2.petals.se.talend.TalendSe</jbi:component-class-name>
       <jbi:component-class-path>
           <jbi:path-element/>
       </jbi:component-class-path>

       <jbi:bootstrap-class-name>org.ow2.petals.component.framework.DefaultBootstrap</jbi:bootstrap-class-name>
       <jbi:bootstrap-class-path>
           <jbi:path-element />
       </jbi:bootstrap-class-path>

       
       <petalsCDK:acceptor-pool-size>5</petalsCDK:acceptor-pool-size>
       <petalsCDK:processor-pool-size>10</petalsCDK:processor-pool-size>
       <petalsCDK:ignored-status>DONE_AND_ERROR_IGNORED</petalsCDK:ignored-status>
       <petalsCDK:jbi-listener-class-name>org.ow2.petals.se.talend.TalendJBIListener</petalsCDK:jbi-listener-class-name>

       
       
       <talend:validate-exchange-by-wsdl>false</talend:validate-exchange-by-wsdl>

   </jbi:component>
</jbi:jbi>{code}

\\
The component configuration includes the configuration of the CDK. The following parameters correspond to the CDK configuration.
{include:0 CDK Component Configuration Table}

\\
This component also has one specific configuration parameter.
|| Parameter || Description || Default || Required ||
| validate-exchange-by-wsdl | True to validate the received messages with respect to the WSDL of the target service. \\ |

This parameter is also available in the configuration of the service-units.
\\
Setting it in the component enables it for all the service-units deployed on this component. It also overrides the service-unit configuration for this parameter.
\\
\\
Beware, for the moment, WSDL-based validation does not work with messages having attachments. \| false \| false \|

\\
{warning}
The Petals-SE-Talend component can only handle messages coming from inside the bus. Therefore, you cannot specify an external-listener class-name.{warning}

h1. Service Configuration

h2. Service Unit descriptor

The Service Unit descriptor file ( jbi.xml ) looks like this:
{code:lang=xml}
<?xml version="1.0" encoding="UTF-8"?>
<jbi:jbi version="1.0"
   xmlns:jbi="http://java.sun.com/xml/ns/jbi"
   xmlns:petalsCDK="http://petals.ow2.org/components/extensions/version-5"
   xmlns:talend="http://petals.ow2.org/components/talend/version-1"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

   <jbi:services binding-component="false">
       <jbi:provides
           interface-name="generatedNs:AttachmentInOutServicePortType"
           service-name="generatedNs:AttachmentInOutService"
           endpoint-name="AttachmentInOutEndpoint"
           xmlns:generatedNs="http://petals.ow2.org/talend/">

           
           <petalsCDK:wsdl>AttachmentInOut.wsdl</petalsCDK:wsdl>
           <petalsCDK:validate-wsdl>true</petalsCDK:validate-wsdl>

           
           <talend:name>AttachmentInOut</talend:name>
           <talend:class-name>talenddemosjava.attachmentinout_0_1.AttachmentInOut</talend:class-name>
           <talend:context>Default</talend:context>
           <talend:singleton>true</talend:singleton>
           <talend:validate-exchange-by-wsdl>false</talend:validate-exchange-by-wsdl>


           <talend:output-attachment>outputFile</talend:output-attachment>
       </jbi:provides>
   </jbi:services>
</jbi:jbi>

{code}

\\
*Configuration of a Service Unit to expose a Talend job as a service into Petals ESB :*

|| Parameter || Description || Default || Required ||
| name | The job's name \\ | \- | Yes |
| class-name | The job's class name \\ | \- | Yes |
| context | The context to use (if invalid, the job will use the default context) \\ | \- | Yes |
| singleton | The singleton property \\ | \- | Yes |
| validate-exchange-by-wsdl | Validate the messages against the WSDLs' schemas | False | No \\ |
| output-attachment \\ | The name of a context variable that points to a file that must be attached to the returned message. \\
The cardinality for this element is 0-*. \\ | \- | No \\ |

\\
{include:0 CDK SU Provide Configuration}
\\
{include:0 CDK Interceptor configuration for SU}

h2. Service Unit content 

The service unit must contain the JAR archives of the job and its dependencies, plus JAR file containing all the contexts..
It is also highly recommended to provide a WSDL description of your job. This WSDL is not mandatory, but not providing it will prevent your service from interacting with other Petals services and components.
By default, a WSDL is generated during the Talend export for Petals.

\\
The directory structure of a SU for the Petals-SE-Talend looks like this:

{noformat}su-talend-JobName-provide.zip
* META-INF
** jbi.xml
* src
** { source files }
* JobName.wsdl
* contexts.jar
* systemRoutines.jar
* userRoutines.jar
* JobName_JobVersion.jar
* JobDependencies.jar (there can be several jars){noformat}

Wiki

All versions of Petals ESB

Helpful links

Petals-SE-Talend

Changes (1)