Define New Transformer with Tcl Commands

This message appears in the splash window of FME Workbench: "Loading Transformers..."

FME Transformers are defined in "*.fmx" files; the Workbench is loading the definitions from the files saved in these folders.
<FME HOME>\transformes
C:\Users\<username>\Documents\FME\Transformers
=====
2014-10-12: And also here, if there are transformers downloaded from the FME Store.
C:\Users\<username>\AppData\Roaming\Safe Software\FME\FME Store\Transformers
=====

If you define a transformer with the correct syntax and save it to one of these folders as "*.fmx" file, it will be added to the Transformer Gallery after restarting Workbench.
=====
Alternatively, click the "refresh" button of Transformer Gallery to reload transformer definitions. It was pointed out by Dale@Safe. See also his comment.
=====
There are several ways to define a transformer. Although I don't know everything, I recently learned about a way to define a feature-based transformer using Tcl commands.
A Tcl-based transformer can be defined with a text editor such as Notepad.

Example: AttributeMultiplier Version 0

Download "AttributeMultiplier_v0.fmx", save it to one of the folders mentioned above, and restart Workbench (or click the "refresh" button of Transformer Gallery). Then the transformer will be added to the Transformer Gallery.
This transformer multiplies every specified attribute by a value that you specified to the "Multiplier" parameter. If the original attribute value is non-numeric, this transformer does nothing against the attribute. i.e. leaves the original value.

Open the fmx file with a text editor to see the definition. And see these documentations to learn more about the syntax.
Transformer Definition Documentation
FME GUI Type Documentation
TclCaller (FME Tcl Variables and Functions)

You can also see the definitions of existing transformers in this file as good examples.
But be careful not to destroy it. I cannot support you to restore the file ;)
<FME HOME>\transformes\fmesuite.fmx

Example: AttributeMultiplier Version 1

Upgrade is easy. Just create new definition with new version number which is greater than the previous one. You can write the new definition in either the same file or another new file.
=====
2014-09-29: I found that two same transformer names appear in the Transformer Gallery when two files exist for two versions simultaneously, although both of them refer to the new version. So, it seems better to insert the new definition before the old definition in the same file if you need to retain the old version after upgrading.
=====
The version 1 example is here: "AttributeMultiplier_v1.fmx"
In the version 1, you can choose one of these actions which should be performed against non-numeric attributes.
- Do nothing
- Replace with a value that you specified to the "New Value" parameter
- Replace with an empty string
- Replace with <null>

Interesting?
To uninstall the transformer, just remove the fmx file.

FME 2014 SP3 build 14391

P.S. I wanted to create a transformer that will hide specified attributes/lists. The functionality should be same as "Attributes to Hide" and "Lists to Hide" in the PythonCaller. Lists were able to be hidden, but I have not achieved a way to hide attributes yet. It seems not to be easy...

FME 2015: Advanced Text Editor Improvement for International Characters

The representation of a Japanese word usually consists of "Kanji" and/or "Hiragana" characters.
A "Kanji" represents a meaning of something. Its pronunciation consists of one or more syllables. Many "Kanji" have two or more different pronunciations depending on the context. And also, there can be several different "Kanji" for the same pronunciation.
A "Hiragana" basically represents a syllable. e.g. a, i, u, e, o, ka, ki, ku, ke, ko.
In fact, there is another character type called "Katakana" for representing syllables, but I don't mention about it here.
You feel so troublesome? But I have to use mixing the Japanese character set and the ASCII characters also in FME workspaces.

In a general text editor such as Notepad, you can enter a Japanese word with these steps in Japanese input mode.
1. Enter the word with English characters according to the pronunciation. The characters will be converted to "Hiragana" automatically on the screen, one by one syllable.
2. After entering entire syllables of the word, push the Space key to convert it to a candidate representation including "Kanji".
3. If the candidate is the preferable one, push the Enter key to decide the input.
4. If not, push the Space key again to display a list of candidate representations, and select the preferable one.

In all the parameter fields of FME Workbench, you can enter Japanese words with the same way above.

FME 2014 and earlier
However, the Advanced Text Editor in FME 2014 (and earlier) does not display characters while entering them in the step 1 and 2. Although the list of candidates will appear after entering entire syllables and pushing the Space key (step 4), it's too inconvenient since there could be a typo while entering, and it causes mis-conversions easily.
So I often enter a Japanese word in an external text editor and then copy-and-paste it to the Advanced Text Editor. It's also troublesome, but is a little more efficient than the "blind-entering" in some cases.

FME 2015
I found that some enhancements for the editor have been done in FME 2015, so that the "blind-entering" has been resolved.
The new Advanced Text Editor displays the characters as same as Notepad, so I can enter Japanese characters directly and efficiently. The usability became much higher.
I believe that the improvement brings a big benefit also to every character set which needs a specific character input mode. e.g. Chinese Hanzi, Korean Hangul.

Anyway, I had been waiting for this for long time.
Thanks for the great improvement!

FME 2015 beta build 15179

How to Specify List Name through Custom Transformer Parameter

In many existing transformers, users can specify name of list attribute that will be created by the transformer. e.g. ListBuilder, AttributeSplitter, StringSearcher etc.
As far as I know, the ListRenamer is the only way to add an interface to a custom transformer, so that users can specify list name, like the existing transformers.

Custom transformer implementation:
1) Create a published user parameter to receive list name specified by the user.
2) Add a ListRenamer to rename a target list to the parameter value.

Then, users can specify the list name through the parameter.
FME 2014 SP3 build 14391

=====
2014-09-30: This is a workspace that contains a custom transformer example which renames attribute/list based on parameter values specified by the user. FME 2014 SP3+
MyRenamer_FME2014.fmw
=====

In FME 2015 beta build 15179, such a custom transformer created with FME 2014 works fine to rename list, but the default name ("_list" in this example) is always visible in the Main Canvas.
FME 2015 Beta build 15179

Result of the translation is fine, but the GUI is not fun...

And also, I found that much enhancements have been done for the ListRenamer itself in FME 2015 beta. All the enhancements are great, but a user parameter cannot be used as new list name, at least in build 15179.
I hope one more enhancement for that, if possible.

=====
2014-11-13: I found another trick by chance in the implementation of the StringMultiSearcher (FME Store transformer). This screenshot illustrates the essence of the trick.

Transformers in the Bookmark expose list names specified by Published Parameters named "listName1" and "listName2" (default values are "_list1" and "_list2"). ​They do nothing other than the exposing; actual list attributes will be created by the PythonCaller.
The upper stream in the Bookmark exposes a structured list name:
$(listName1){}.sub1
$(listName1){}.sub2
The lower stream exposes a simple list name:
$(listName2){}
There are no inputs for the streams, it looks too strange but worked fine as expected in both FME 2014 SP4 build 14433 and FME 2015 Beta build 15210.

In addition, I tried using ListBuilders instead of the Aggregators so that the AttributeRemover_2 can be removed.
FME 2014: It worked fine with no problems.
FME 2015: It also worked fine but the default list names have appeared again in the interface of the custom transformer...orz
I'm not sure what has caused the difference.
Stop the investigation here in order not to fall into the FME darkness.

Replace String using Regular Expression {.*}

Python re.sub method:
-----
>>> import re
>>> re.sub('.*', 'bar', 'foo')
'bar'
-----

Tcl regsub command:
-----
% regsub -all {.*} foo bar
barbar

-----

I feel the result of the Tcl expression is a little bit strange. If the regex was {^.*} or {^.*$}, the expression returns bar, but {.*$} returns the same result above - barbar.

How do you think about it?

Anyway, FME seems to be using Tcl regsub command internally for string replacement operations in some parts. e.g. StringReplacer (Use Regular Expressions: yes), BulkAttributeRenamer (Action: Regular Expression Replace), etc.

Maybe we will have to check carefully the replacing result with {.*}, especially in FME 2015.

Why not publish Parallel Processing parameters for the Custom Transformer?

Maybe someone is wondering why I didn't publish Parallel Processing parameters in the RandomPointGenerator implementation (see the previous post).

If you publish the "Parallel Process By" parameter (Right-click > Create User Parameter), you can make the transformer as a block-based one easily. i.e. the parameter works like "Group By".

Originally I had published the parameter while developing the transformer, but I noticed that this warning message appears on the Log window after the translation.
The message always appears if the workspace has the following conditions.
-----
Warning: not all FMESessions that were created were destroyed before shutdown. This may cause instability
-----
Conditions:
1) The custom transformer contains a StatisticsCalculator.
2) The Main workspace containing the custom transformer also contains a StatisticsCalculator.
3) Specify attribute(s) to the "Parallel Process By" parameter of the custom transformer.
4) Select "No Parallelism" (default) for the "Parallel Processing Level" parameter of the custom transformer.
-----
FME 2014 SP3 build 14391

I know that the warning may not cause any problem if the workspace is running with FME Desktop. But I'm afraid that it could cause a serious problem if the workspace is running with FME Server. And also I cannot confirm whether there are any other conditions which cause the same situation.

Therefore, I will not publish Parallel Processing parameters for a custom transformer unless the issue is resolved.

P.S. The same warning has occurred when using the AttributePivoter. These two transformers - StatisticsCalculator and AttributePivoter are implemented with Python. I guess there is an issue in the architecture of Python based transformers.

RandomPointGenerator

I created a custom transformer named RandomPointGenerator, which creates random points within entire input area features.
Find the transformer in Transformer Gallery (FME Store / Pragmatica).
FME 2014 SP3 build 14391

The implementation has been inspired by this thread in FME Community Answers.
Stratified random point in FME

Here, I illustrate outline of the procedure.

1. Input areas example.
Assume that these areas belong to a same group; these could be parts of an aggregated feature; there may be overlapped areas.

2. Resolve the overlapped area; calculate area for every area.

Deaggregator
AreaOnAreaOverlayer
AreaCalculator







3. Create bounding boxes for each individual area.

BoundingBoxReplacer









4. Dissolve the boxes, and create bounding boxes again; calculate area for every box.

Dissolver
BoundingBoxReplacer
AreaCalculator







5. Compute preferable number of points for each box  (based on the area calculation results and the required number of points), and then randomly create points within the boxes, so that the density will be approximately identical.

StatisticsCalculator
BoundsExtractor
AttributeCreator
ExpressionEvaluator
VertexCreator
Cloner
Offsetter



6. Filter the points by the original areas.

SpatialFilter









7. Randomly take points of specified number.

ExpressionEvaluator
Sorter
Sampler

Concatenate Attribute Values based on Attribute List Parameter

A custom transformer can receive space delimited attribute names through "Attribute List (space delimited)" type user parameter.

Typically, a parameter of this type would be used to link to "Group By" of transformer(s) in the custom transformer. In such a case, it's not necessary to retrieve values of the attributes.
But there could also be some cases where those values are necessary. e.g. use concatenated attribute values as Counter Name of a Counter.

Value of the parameter is a string. If specified attributes are "attr1", "attr2", and "attr3", for example, the value of parameter will be "attr1 attr2 attr3" (space delimited attribute names). And if an attribute name contains white spaces, it will be quoted by double quotations.

I think Tcl scripting is an easy way to concatenate specified attribute values.
First, use a ParameterFetcher to store the parameter value (i.e. specified attribute names) as an attribute. e.g. named "_attr_names".
And then, use a TclCaller with this script to create a concatenated string.
If no attributes were specified, this procedure returns an empty string.
-----
proc concatAttributeValues {} {
    set values {}
    foreach attr [FME_GetAttribute "_attr_names"] {
        lappend values [FME_GetAttribute $attr]
    }
    return [join $values {_}]
}
-----

It worked fine even if specified attribute names contain white spaces.

FME 2014 SP3 build 14391

Computational Error in DMSCalculator

The DMSCalculator is useful to convert decimal degree value to degrees, minutes and seconds. But there may be cases where the slight computational error causes a trouble.

=====
2014-10-03: I had discussed about the computational error in the DMSCaluculator (Version 0) with Safe's developer, then the new DMSCaluclator (Version 1) has become available in FME 2015 Beta build 15187. It's better on handling the computational error than the previous version, you will not need to consider the following workaround, in FME 2015.
=====
Although computational error cannot be avoided, there are ways to reduce it in some cases.
In the DMS calculation, I prefer to define math expressions myself, rather than using the DMSCalculator, so that the computational error can be reduced.
For example:

1) ExpressionEvaluator

2) AttributeCreator
=====
2014-09-24: Sorry, I didn't take care of negative source values - i.e. south/west hemisphere latitude/longitude. I'm living in north/east hemisphere of the earth.
Try these expressions for calculating minutes and seconds, if the source may be negative.
When the source is negative, a minus sign will be added to degrees value only.
-----
_minutes: @int(@abs(@Value(_seconds)))%3600/60
_seconds: @fmod(@abs(@Value(_seconds)),60)
=====

3) Result

(FME 2014 SP3 build 14391)

Transform 3D Triangular Polygons into TIN Surface

Assume that the source polygons are clean. i.e.
- There are no gaps and no overlaps between adjoining triangles.
- Common vertex of adjoining triangles has same elevation (z-coordinate).

There is a solution in this article.
How to clip TIN Surfaces

Summary:
1) FaceReplacer replaces the triangular polygons with faces.
2) Aggregator creates a multi-surface from the faces.
3) Triangulator transforms the multi-surface into a TIN surface.
If orientation of triangles could be Right Hand Rule, their orientation should be changed to Left Hand Rule before the FaceReplacer.

The purpose of the article is not to create new TIN surface, is to clip a given TIN surface. So, the solution is appropriate one. But it's not so efficient for newly creating a TIN surface based on millions triangular polygons.
If you allow that the original triangle shapes will not be preserved exactly as parts of the resulting TIN, this may be a much more efficient approach.

1) Chopper decomposes the triangles into vertex points.
2) CoordinateExtractor extracts x, y of each point.
3) AttributeRounder rounds x, y to avoid mismatching caused by slight computational error.
4) StringConcatenator concatenates x, y to create a string value.
5) DuplicateRemover removes duplicate points using the coordinate string as key.
6) TINGenerator creates a TIN surface based on the points.
In general, the Matcher can be used to remove duplicate points. But it's too inefficient (consumes long time and huge memory space) if there are millions points.

There could be more efficient solutions. If you know, please inform to me!

FME 2014 SP3 build 14391

Split Polygons Retaining Attributes

I needed to split given land use polygons (Esri Shape format) according to actual land usage which can be guessed based on images taken by a satellite.

At first, I planned to split the polygons with the Cut Polygons tool in ArcGIS.
But the client frequently requests to me modifying the cutting lines. Once I cut a polygon with the tool, I have to move 2 lines simultaneously without any gaps to modify the line, since the line immediately becomes boundaries shared by adjoining areas. It's so troublesome.

Therefore, I stopped using the Cut Polygons tool. I decided to create a new layer which represent cutting lines using ArcMap, and create split polygons as new dataset using FME.

Splitting polygons is easy. Intersect the polygons with the cutting lines, and build areas from the Intersected lines.

But the step cannot retain attributes of the original polygons. To retain attributes, migrate attributes of the original polygons based on spatial relationships.

Modifications of the cutting lines became much easier by this method.
FME 2014 SP3 build 14391