Previous: Startup / Shutdown Script
A list attribute contains multiple values under the same name, each element of a list is distinguished from others by its index. The index is usually 0-based sequential number quoted by braces (curly brackets); it's a part of the attribute name indicating an element.
Names of individual list elements are typically represented like these:
_list{0}, _list{1}, _list{2} ...
_list{0}.sub, _list{1}.sub, _list{2}.sub ...
Every element can be treated as well as a non-list attribute using its name (including index) in the Tcl script. And also, since the name contains an index, repetitive processing commands (for, foreach etc.) would be effective in many cases.
Here I'll give some Tcl procedure examples for the TclCaller.
Convert Multiple Attributes to a List Attribute
This procedure collects 3 non-list attribute values and create a list attribute named _code which contains original 3 attribute values as its elements. This functionality is similar to the ListPopulator transformer.
-----
proc multiAttributesToList {} {
set i 0
foreach name {"CODE_A" "CODE_B" "CODE_C"} {
FME_SetAttribute "_code{$i}" [FME_GetAttribute $name]
incr i
}
}
-----
Concatenate List Elements
This procedure concatenates every element in a list attribute named _list, and returns the result as a comma separated string. Similar to the ListConcatenator transformer.
-----
proc concatList {} {
set elements {}
for {set i 0} {[FME_AttributeExists "_list{$i}"]} {incr i} {
lappend elements [FME_GetAttribute "_list{$i}"]
}
return [join $elements {,}]
}
-----
Calculate Basic Statistics of List Elements
This procedure calculates sum, minimum, maximum and average of elements of a list attribute named _src, sets the results as new non-list attributes, and returns number of the elements. Assume all elements of the list are numeric representation.
-----
proc calcBasicStatistics {} {
for {set count 0} {[FME_AttributeExists "_src{$count}"]} {incr count} {
set val [FME_GetAttribute "_src{$count}"]
if {$count == 0} {
set sum [set min [set max $val]]
} else {
set sum [expr $sum + $val]
if {$val < $min} {set min $val}
if {$max < $val} {set max $val}
}
}
if {0 < $count} {
FME_SetAttribute "_sum" $sum
FME_SetAttribute "_min" $min
FME_SetAttribute "_max" $max
FME_SetAttribute "_ave" [expr double($sum) / $count]
}
return $count
}
-----
"Getting Started with Tcl in FME" series ends here. My Tcl skill made progress a little???
"Getting Started with Tcl in FME" Titles
TclCaller
AttributeCreator
Other Transformers
Scripted Parameter
Startup / Shutdown Script
List Attribute Manipulation in TclCaller *this post
2013-10-30
Advanced Geometric Operation: Divide a Line at Equal Interval
There are many transformers for geometric operations in FME Workbench; one of them or combination of some transformers can do most of all geometric operations we need usually. But sometimes, I encounter a case which needs to create a custom transformer or Python scripting.
I think that dividing a line at equal interval is one of such cases, and also that is relatively frequent required operation.
The Snipper transformer can be used to divide a line. However, it divides a line into only two parts (snipped and remnant) per one processing. If I set the length of the snipped part as the specified interval, the dividing operation has to be performed repeatedly while the length of the remnant part is greater than specified interval.
So, I think a custom transformer having a loop is an effective solution to do that.
This screenshot is a simplified example without any validation against input features.
Note: Actually the LengthCalculator and the Tester may not be essential. But when lacking them, the Workbench will display warning messages.
=====
2013-11-25
It is a kind of "reinventing the wheel". FME Store provides the IterativeSnipper_2013 custom transformer. Look at this as an exercise for the loop functionality of a custom transformer.
=====
Similar operation can be also defined as Python script for the PythonCaller.
-----
# Example: Divide Line at Equal Interval
# 2013-11-27 Modified
import fmeobjects
class LineDivider(object):
def __init__(self):
self.geomCloner = fmeobjects.FMEFeature()
def input(self, feature):
try:
interval = float(feature.getAttribute('_interval'))
if interval < 1.0e-6:
raise fmeobjects.FMEException()
dim = int(feature.getAttribute('_length_dimension'))
measure3D = (True if dim == 3 else False)
remnant = feature.getGeometry()
if isinstance(remnant, fmeobjects.FMESimpleArea):
remnant = remnant.getBoundaryAsCurve()
curveList = []
while interval < remnant.getLength(measure3D):
self.geomCloner.setGeometry(remnant)
curve = self.geomCloner.getGeometry()
curve.snip(fmeobjects.SNIP_DISTANCE, measure3D, 0, interval)
curveList.append(curve)
remnant.snip(fmeobjects.SNIP_DISTANCE, measure3D, interval, -1)
if 0.0 < remnant.getLength(measure3D):
curveList.append(remnant)
for i, geom in enumerate(curveList):
feat = feature.cloneAttributes()
feat.setGeometry(geom)
feat.setAttribute('_index', i)
feat.setAttribute('_result', 'success')
feat.setCoordSys(feature.getCoordSys())
self.pyoutput(feat)
except:
feature.setAttribute('_result', 'fail')
self.pyoutput(feature)
def close(self):
pass
-----
Next: Effective Cloner
I think that dividing a line at equal interval is one of such cases, and also that is relatively frequent required operation.
The Snipper transformer can be used to divide a line. However, it divides a line into only two parts (snipped and remnant) per one processing. If I set the length of the snipped part as the specified interval, the dividing operation has to be performed repeatedly while the length of the remnant part is greater than specified interval.
So, I think a custom transformer having a loop is an effective solution to do that.
This screenshot is a simplified example without any validation against input features.
Note: Actually the LengthCalculator and the Tester may not be essential. But when lacking them, the Workbench will display warning messages.
=====
2013-11-25
It is a kind of "reinventing the wheel". FME Store provides the IterativeSnipper_2013 custom transformer. Look at this as an exercise for the loop functionality of a custom transformer.
=====
Similar operation can be also defined as Python script for the PythonCaller.
-----
# Example: Divide Line at Equal Interval
# 2013-11-27 Modified
import fmeobjects
class LineDivider(object):
def __init__(self):
self.geomCloner = fmeobjects.FMEFeature()
def input(self, feature):
try:
interval = float(feature.getAttribute('_interval'))
if interval < 1.0e-6:
raise fmeobjects.FMEException()
dim = int(feature.getAttribute('_length_dimension'))
measure3D = (True if dim == 3 else False)
remnant = feature.getGeometry()
if isinstance(remnant, fmeobjects.FMESimpleArea):
remnant = remnant.getBoundaryAsCurve()
curveList = []
while interval < remnant.getLength(measure3D):
self.geomCloner.setGeometry(remnant)
curve = self.geomCloner.getGeometry()
curve.snip(fmeobjects.SNIP_DISTANCE, measure3D, 0, interval)
curveList.append(curve)
remnant.snip(fmeobjects.SNIP_DISTANCE, measure3D, interval, -1)
if 0.0 < remnant.getLength(measure3D):
curveList.append(remnant)
for i, geom in enumerate(curveList):
feat = feature.cloneAttributes()
feat.setGeometry(geom)
feat.setAttribute('_index', i)
feat.setAttribute('_result', 'success')
feat.setCoordSys(feature.getCoordSys())
self.pyoutput(feat)
except:
feature.setAttribute('_result', 'fail')
self.pyoutput(feature)
def close(self):
pass
-----
Next: Effective Cloner
2013-10-27
Specify Log File Name
By default, the log file name will be same as the workspace name, except its extension. If we need to change the log file name, can specify a preferable name to a Workspace Parameter named "Log File". For example:
$(FME_MF_DIR)MyLogFile.log
FME_MF_DIR is a macro (system parameter) name, indicates the current directory path. "./MyLogFile.log" can be also specified here.
But the log file name specified with such a way is fixed, so the file will be overwritten when the workspace is executed multiple times repeatedly. When we need to preserve every log file with unique name for each run, how can we do that?
1. Rename in the Shutdown Script
One possible approach is to rename the log file name in the Shutdown Script.
Note:This approach cannot be applied in a certain situation as mentioned later.
The log file has been closed before shutdown process; we can access the saved log file path via a global variable named "FME_LogFileName" in the script.
This Shutdown Python Script appends a timestamp (and sequential number when the timestamp also conflicts) to the log file name.
-----
import os, time, re
src = FME_LogFileName
if src and os.path.exists(src):
stamp = time.strftime('%Y%m%d_%H%M%S')
name = '%s_%s' % (re.sub('^(.+)\\.log$', '\\1', src), stamp)
base = name
count = 1
while os.path.exists('%s.log' % name):
name = '%s_%d' % (base, count)
count += 1
os.rename(src, '%s.log' % name)
-----
2. Define a Scripted Parameter
If the translation process is executed always one by one, the renaming approach works well. However, if two or more processes for the same workspace will be executed simultaneously - e.g. when the WorkspaceRunner specified "No" to "Wait for Job Complete" runs the workspace several times, we cannot get the expected result. Because the original log file name conflicts between the processes.
In such a situation, we should create a unique log file name as a scripted parameter, and link the Workspace Parameter "Log File" to it. The script can be like this, for example.
-----
import os, time, re
src = FME_MacroValues['FME_MF_DIR'] + FME_MacroValues['FME_MF_NAME']
stamp = time.strftime('%Y%m%d_%H%M%S')
name = '%s_%s' % (re.sub('^(.+)\\.fmw$', '\\1', src), stamp)
base = name
count = 1
while os.path.exists('%s.log' % name):
name = '%s_%d' % (base, count)
count += 1
return '%s.log' % name
-----
Appendix: Tcl Script Examples
The following Tcl scripts work as well as the Python examples above.
Shutdown Tcl Script: Rename the Log File Name
------
set src $FME_LogFileName
if {[string length $src] && [file exists $src]} {
set stamp [clock format [clock seconds] -format {%Y%m%d_%H%M%S}]
set name [format "%s_%s" [regsub {^(.+)\.log$} $src {\1}] $stamp]
set base $name
for {set count 1} {[file exists [format "%s.log" $name]]} {incr count} {
set name [format "%s_%d" $base $count]
}
file rename $src [format "%s.log" $name]
}
------
Scripted (Tcl) Parameter: Create a Unique Log File Name
------
set src "$FME_MacroValues(FME_MF_DIR)$FME_MacroValues(FME_MF_NAME)"
set stamp [clock format [clock seconds] -format {%Y%m%d_%H%M%S}]
set name [format "%s_%s" [regsub {^(.+)\.fmw$} $src {\1}] $stamp]
set base $name
for {set count 1} {[file exists [format "%s.log" $name]]} {incr count} {
set name [format "%s_%d" $base $count]
}
return [format "%s.log" $name]
------
See another example here:
> Community: FME -> log file
$(FME_MF_DIR)MyLogFile.log
FME_MF_DIR is a macro (system parameter) name, indicates the current directory path. "./MyLogFile.log" can be also specified here.
But the log file name specified with such a way is fixed, so the file will be overwritten when the workspace is executed multiple times repeatedly. When we need to preserve every log file with unique name for each run, how can we do that?
1. Rename in the Shutdown Script
One possible approach is to rename the log file name in the Shutdown Script.
Note:This approach cannot be applied in a certain situation as mentioned later.
The log file has been closed before shutdown process; we can access the saved log file path via a global variable named "FME_LogFileName" in the script.
This Shutdown Python Script appends a timestamp (and sequential number when the timestamp also conflicts) to the log file name.
-----
import os, time, re
src = FME_LogFileName
if src and os.path.exists(src):
stamp = time.strftime('%Y%m%d_%H%M%S')
name = '%s_%s' % (re.sub('^(.+)\\.log$', '\\1', src), stamp)
base = name
count = 1
while os.path.exists('%s.log' % name):
name = '%s_%d' % (base, count)
count += 1
os.rename(src, '%s.log' % name)
-----
2. Define a Scripted Parameter
If the translation process is executed always one by one, the renaming approach works well. However, if two or more processes for the same workspace will be executed simultaneously - e.g. when the WorkspaceRunner specified "No" to "Wait for Job Complete" runs the workspace several times, we cannot get the expected result. Because the original log file name conflicts between the processes.
In such a situation, we should create a unique log file name as a scripted parameter, and link the Workspace Parameter "Log File" to it. The script can be like this, for example.
-----
import os, time, re
src = FME_MacroValues['FME_MF_DIR'] + FME_MacroValues['FME_MF_NAME']
stamp = time.strftime('%Y%m%d_%H%M%S')
name = '%s_%s' % (re.sub('^(.+)\\.fmw$', '\\1', src), stamp)
base = name
count = 1
while os.path.exists('%s.log' % name):
name = '%s_%d' % (base, count)
count += 1
return '%s.log' % name
-----
Appendix: Tcl Script Examples
The following Tcl scripts work as well as the Python examples above.
Shutdown Tcl Script: Rename the Log File Name
------
set src $FME_LogFileName
if {[string length $src] && [file exists $src]} {
set stamp [clock format [clock seconds] -format {%Y%m%d_%H%M%S}]
set name [format "%s_%s" [regsub {^(.+)\.log$} $src {\1}] $stamp]
set base $name
for {set count 1} {[file exists [format "%s.log" $name]]} {incr count} {
set name [format "%s_%d" $base $count]
}
file rename $src [format "%s.log" $name]
}
------
Scripted (Tcl) Parameter: Create a Unique Log File Name
------
set src "$FME_MacroValues(FME_MF_DIR)$FME_MacroValues(FME_MF_NAME)"
set stamp [clock format [clock seconds] -format {%Y%m%d_%H%M%S}]
set name [format "%s_%s" [regsub {^(.+)\.fmw$} $src {\1}] $stamp]
set base $name
for {set count 1} {[file exists [format "%s.log" $name]]} {incr count} {
set name [format "%s_%d" $base $count]
}
return [format "%s.log" $name]
------
See another example here:
> Community: FME -> log file
2013-10-26
Remove Attribute to Write Null
Some data formats - typically databases distinguish null value and empty string, and null value could be important sometimes. But FME 2013 doesn't support null value.
When we need to write a null value into the destination dataset, should remove the attribute beforehand. The writer will treat the value of a missing (removed) attribute as null.
=====
2012-12-23 The NullAttributeMapper transformer has been added in FME 2014.
It can be used to convert between non-null and <null> effectively.
> Null in FME 2014: Converting Null to Non-Null
=====
In general, the AttributeRemover can be used to remove attributes. And also the TclCaller or the PythonCaller could be effective in certain cases.
For example, the following scripts remove every empty attribute.
TclCaller Script Example
-----
proc removeEmptyAttributes {} {
foreach name [FME_AttributeNames] {
if {[string compare [FME_GetAttribute $name] ""] == 0} {
FME_UnsetAttributes $name
}
}
}
-----
PythonCaller Script Example
-----
import fmeobjects
def removeEmptyAttributes(feature):
for name in feature.getAllAttributeNames():
if str(feature.getAttribute(name)) == '':
feature.removeAttribute(name)
-----
Note: When an attribute can hold a unicode object, the str function in the Python script might throw an error saying <UnicodeEncodeError>. See here how to avoid it:
FME stores all attributes as character strings: Part 2
FME 2014 will support null value. I'm looking forward to it.
FMEpedia > How does FME handle null attribute values?
Community > NULL and empty strings in FME 2013
When we need to write a null value into the destination dataset, should remove the attribute beforehand. The writer will treat the value of a missing (removed) attribute as null.
=====
2012-12-23 The NullAttributeMapper transformer has been added in FME 2014.
It can be used to convert between non-null and <null> effectively.
> Null in FME 2014: Converting Null to Non-Null
=====
In general, the AttributeRemover can be used to remove attributes. And also the TclCaller or the PythonCaller could be effective in certain cases.
For example, the following scripts remove every empty attribute.
TclCaller Script Example
-----
proc removeEmptyAttributes {} {
foreach name [FME_AttributeNames] {
if {[string compare [FME_GetAttribute $name] ""] == 0} {
FME_UnsetAttributes $name
}
}
}
-----
PythonCaller Script Example
-----
import fmeobjects
def removeEmptyAttributes(feature):
for name in feature.getAllAttributeNames():
if str(feature.getAttribute(name)) == '':
feature.removeAttribute(name)
-----
Note: When an attribute can hold a unicode object, the str function in the Python script might throw an error saying <UnicodeEncodeError>. See here how to avoid it:
FME stores all attributes as character strings: Part 2
FME 2014 will support null value. I'm looking forward to it.
FMEpedia > How does FME handle null attribute values?
Community > NULL and empty strings in FME 2013
2013-10-22
Getting Started with Tcl in FME: Startup / Shutdown Script
Previous: Scripted Parameter
Startup Tcl Script Example: Log the translation beginning time
-----
# "timeStamp" procedure
# This procedure gets the current system time,
# and returns formatted date time string (time stamp).
proc timeStamp {} {
clock format [clock seconds] -format "%Y-%m-%d %H:%M:%S"
}
# Save the translation beginning time stamp.
set begTime [timeStamp]
# Show and log the translation beginning time.
FME_LogMessage fme_inform "Translation started at $begTime."
-----
Tcl Commands > clock
In the Startup Script (and the TclCaller), FME pre-defined procedure FME_LogMessage can be used to show a message on the Log window and log it into the log file.
Shutdown Tcl Script Example: Log the translation beginning / ending time
-----
# Save beginning and ending time stamp,
# using global variable and procedure defined in the Startup Script.
set begEndTimeStamp "Beg: $begTime\nEnd: [timeStamp]"
# Show the time stamp on the Log window.
puts $begEndTimeStamp
# Append the time stamp to the log file.
set file [open $FME_LogFileName a]
puts $file $begEndTimeStamp
close $file
-----
Tcl Commands > puts, open, close
Global variables and procedures defined in the Startup Script can be used in the Shutdown Script. But those cannot be used in the TclCaller. This is a different from Python script.
Be aware that FME_LogMessage is invalid in the Shutdown Script because the log file will be closed before Shutdown process.
In the Shutdown Script, use "puts" command to show a message on the Log window; to append a message to the log file, it's necessary to re-open the log file with appending mode and write the message into it.
FME_LogFileName is a global variable defined by FME, it holds the log file path.
In fact, FME prepares global variables named FME_StartingTimeStamp and FME_EndingTimeStamp to hold time stamps, so the first line of the example can be replaced with this line:
-----
set begEndTimeStamp "Beg: $FME_StartingTimeStamp\nEnd: $FME_EndingTimeStamp"
-----
Here is the information about global variables defined by FME:
> FMEpedia: Startup and Shutdown Python and Tcl Scripts
And more examples:
> FMEpedia: Startup and Shutdown TCL Script Examples
Next: List Attribute Manipulation in TclCaller
Startup Tcl Script Example: Log the translation beginning time
-----
# "timeStamp" procedure
# This procedure gets the current system time,
# and returns formatted date time string (time stamp).
proc timeStamp {} {
clock format [clock seconds] -format "%Y-%m-%d %H:%M:%S"
}
# Save the translation beginning time stamp.
set begTime [timeStamp]
# Show and log the translation beginning time.
FME_LogMessage fme_inform "Translation started at $begTime."
-----
Tcl Commands > clock
In the Startup Script (and the TclCaller), FME pre-defined procedure FME_LogMessage can be used to show a message on the Log window and log it into the log file.
Shutdown Tcl Script Example: Log the translation beginning / ending time
-----
# Save beginning and ending time stamp,
# using global variable and procedure defined in the Startup Script.
set begEndTimeStamp "Beg: $begTime\nEnd: [timeStamp]"
# Show the time stamp on the Log window.
puts $begEndTimeStamp
# Append the time stamp to the log file.
set file [open $FME_LogFileName a]
puts $file $begEndTimeStamp
close $file
-----
Tcl Commands > puts, open, close
Global variables and procedures defined in the Startup Script can be used in the Shutdown Script. But those cannot be used in the TclCaller. This is a different from Python script.
Be aware that FME_LogMessage is invalid in the Shutdown Script because the log file will be closed before Shutdown process.
In the Shutdown Script, use "puts" command to show a message on the Log window; to append a message to the log file, it's necessary to re-open the log file with appending mode and write the message into it.
FME_LogFileName is a global variable defined by FME, it holds the log file path.
In fact, FME prepares global variables named FME_StartingTimeStamp and FME_EndingTimeStamp to hold time stamps, so the first line of the example can be replaced with this line:
-----
set begEndTimeStamp "Beg: $FME_StartingTimeStamp\nEnd: $FME_EndingTimeStamp"
-----
Here is the information about global variables defined by FME:
> FMEpedia: Startup and Shutdown Python and Tcl Scripts
And more examples:
> FMEpedia: Startup and Shutdown TCL Script Examples
Next: List Attribute Manipulation in TclCaller
2013-10-21
Getting Started with Tcl in FME: Scripted Parameter
Previous: Other Transformers
Parameter values in a workspace are static, we cannot change them while translating. But before starting translation, we can define a parameter according to the conditions. That is the Scripted (Tcl) Parameter, the steps are:
1. Right-click "User Parameters > Private Parameters" node in the Navigator window, select "Add Parameter" to open "Add/Edit Parameter" dialog box.
2. Define Type, Name and Value of the parameter:
- Select "Scripted (Tcl)" for "Type".
- Specify the parameter name to "Name".
- Set a Tcl script to "Value".
Note: Scripted (Python) Parameter can be also defined with the similar steps just replacing "Tcl" with "Python" in the step 2.
> FME Workbench > Creating a User Parameter > Type: Scripted (Python) and Scripted (Tcl)
Example: URL of a file in a secured FTP server
FME readers can read files saved in a secured FTP server via this format URL as the source dataset setting.
ftp://<user name>:<password>@<server name>/<directory path>/<file name>
> Access a Secured FTP Site and Download the Dataset
The URL is constructed of 5 elements, i.e. user name, password, server name, directory path and file name.
If all the elements are known and also they are static, the URL can be set directly to the source dataset parameter of the reader. But if one or more elements of them will have to be determined at each run-time of the workspace, parameterizing them will be effective. And also the the scripted parameter could be useful in such a case.
Assuming that these 5 published parameters have been defined:
USERNAME: user name
PASSWORD: password
FTP_SERVER: FTP server name
FTP_DIRECTORY: directory path
FILENAME: target file name
set this script to "Value" of the Scripted (Tcl) Parameter:
-----
set usr $FME_MacroValues(USERNAME)
set pswd $FME_MacroValues(PASSWORD)
set svr $FME_MacroValues(FTP_SERVER)
set dir $FME_MacroValues(FTP_DIRECTORY)
set file $FME_MacroValues(FILENAME)
return "ftp://$usr:$pswd@$svr/$dir/$file"
-----
"FME_MacroValues" is a dictionary containing macro (parameter) values, we can get other parameter values (defined before this parameter) via this dictionary in the script.
The returned value of the script will be the value of the parameter; the dataset parameter of a reader can be linked to the scripted parameter.
For example, when USERNAME = "MyName", PASSWORD = "MyPassword", FTP_SERVER = "MyFtpServer", FTP_DIRECTORY = "MyRoot/MyFolder" and FILENAME = "*.xml", the reader whose dataset parameter is linked to this scripted parameter will read XML file(s) from this URL:
ftp://MyName:MyPassword@MyFtpServer/MyRoot/MyFolder/*.xml
I think the scripted parameter like the example above is useful, but in this case, this reader dataset parameter setting also works naturally.
ftp://$(USERNAME):$(PASSWORD)@$(FTP_SERVER)/$(FTP_DIRECTORY)/$(FILENAME)
There is more than one way to skin a cat. I often come across a "cat" when working with FME. Although I won't kill a real cat...
Next: Startup / Shutdown Script
Parameter values in a workspace are static, we cannot change them while translating. But before starting translation, we can define a parameter according to the conditions. That is the Scripted (Tcl) Parameter, the steps are:
1. Right-click "User Parameters > Private Parameters" node in the Navigator window, select "Add Parameter" to open "Add/Edit Parameter" dialog box.
2. Define Type, Name and Value of the parameter:
- Select "Scripted (Tcl)" for "Type".
- Specify the parameter name to "Name".
- Set a Tcl script to "Value".
Note: Scripted (Python) Parameter can be also defined with the similar steps just replacing "Tcl" with "Python" in the step 2.
> FME Workbench > Creating a User Parameter > Type: Scripted (Python) and Scripted (Tcl)
Example: URL of a file in a secured FTP server
FME readers can read files saved in a secured FTP server via this format URL as the source dataset setting.
ftp://<user name>:<password>@<server name>/<directory path>/<file name>
> Access a Secured FTP Site and Download the Dataset
The URL is constructed of 5 elements, i.e. user name, password, server name, directory path and file name.
If all the elements are known and also they are static, the URL can be set directly to the source dataset parameter of the reader. But if one or more elements of them will have to be determined at each run-time of the workspace, parameterizing them will be effective. And also the the scripted parameter could be useful in such a case.
Assuming that these 5 published parameters have been defined:
USERNAME: user name
PASSWORD: password
FTP_SERVER: FTP server name
FTP_DIRECTORY: directory path
FILENAME: target file name
set this script to "Value" of the Scripted (Tcl) Parameter:
-----
set usr $FME_MacroValues(USERNAME)
set pswd $FME_MacroValues(PASSWORD)
set svr $FME_MacroValues(FTP_SERVER)
set dir $FME_MacroValues(FTP_DIRECTORY)
set file $FME_MacroValues(FILENAME)
return "ftp://$usr:$pswd@$svr/$dir/$file"
-----
"FME_MacroValues" is a dictionary containing macro (parameter) values, we can get other parameter values (defined before this parameter) via this dictionary in the script.
The returned value of the script will be the value of the parameter; the dataset parameter of a reader can be linked to the scripted parameter.
For example, when USERNAME = "MyName", PASSWORD = "MyPassword", FTP_SERVER = "MyFtpServer", FTP_DIRECTORY = "MyRoot/MyFolder" and FILENAME = "*.xml", the reader whose dataset parameter is linked to this scripted parameter will read XML file(s) from this URL:
ftp://MyName:MyPassword@MyFtpServer/MyRoot/MyFolder/*.xml
I think the scripted parameter like the example above is useful, but in this case, this reader dataset parameter setting also works naturally.
ftp://$(USERNAME):$(PASSWORD)@$(FTP_SERVER)/$(FTP_DIRECTORY)/$(FILENAME)
There is more than one way to skin a cat. I often come across a "cat" when working with FME. Although I won't kill a real cat...
Next: Startup / Shutdown Script
2013-10-17
"Join On" Parameter of the FeatureMerger
The FeatureMerger is one of the most frequently used transformers; it can be used to merge attributes of feature(s) (supplier) to other feature(s) (requestor).
The merging condition should be specified as the "Join On" parameter. In many cases, it will be set to attribute names of requester and supplier features. I think this is the most common usage.
However, the current FeatureMerger (FME 2013 SP2 or later) allows various settings other than attribute names to the "Join On" parameter:
Constant
A constant value (including link to a User / System Parameter) can be specified to the "Join On" parameter. This means we can use the FeatureMerger to perform "unconditional merging". If you set the same value (e.g. 1) to both of requester and supplier, it will merge features unconditionally.
We don't need to neither create an attribute holding constant value beforehand nor download the UnconditionalFeatureMerger from the FME Store.
Expression and Conditional Value
Using the String Editor, we can specify not only a simple concatenated attribute values but also more complicated expression with FME functions or Tcl commands to the "Join On" parameter. Furthermore, the Conditional Value settings can be also used.
If we can make full use of them, the number of transformers may be able to be reduced dramatically in some cases.
I think the current FeatureMerger has extremely larger availability than the previous version, against various needs. When I found a practical usage, will post again!
The merging condition should be specified as the "Join On" parameter. In many cases, it will be set to attribute names of requester and supplier features. I think this is the most common usage.
However, the current FeatureMerger (FME 2013 SP2 or later) allows various settings other than attribute names to the "Join On" parameter:
Constant
A constant value (including link to a User / System Parameter) can be specified to the "Join On" parameter. This means we can use the FeatureMerger to perform "unconditional merging". If you set the same value (e.g. 1) to both of requester and supplier, it will merge features unconditionally.
We don't need to neither create an attribute holding constant value beforehand nor download the UnconditionalFeatureMerger from the FME Store.
Expression and Conditional Value
Using the String Editor, we can specify not only a simple concatenated attribute values but also more complicated expression with FME functions or Tcl commands to the "Join On" parameter. Furthermore, the Conditional Value settings can be also used.
If we can make full use of them, the number of transformers may be able to be reduced dramatically in some cases.
I think the current FeatureMerger has extremely larger availability than the previous version, against various needs. When I found a practical usage, will post again!
2013-10-13
Getting Started with Tcl in FME: Other Transformers
Previous: AttributeCreator
=====
2015-02-22 Note: The @Evaluate function of FME 2015 will not return non-numeric value any longer. If the Tcl expression which is embedded to the argument for the function returns non-numeric, the @Evaluate functions will return <null>. Therefore, the following examples are unavailable in FME 2015 unfortunately.
=====
Several transformers other than the TclCaller and the AttributeCreator also accept Tcl commands in the parameter setting. Here I'll give some examples.
FME 2013 SP4 Build 13545 Win32
In the following examples, "string" command is used with various options.
Tcl Commands > string
Tester with this setting decides whether the length (number of characters) of "attr" is less than 10.
@Evaluate([string length {@Value(attr)}]) < 10
TestFilter with this setting routes features into 3 different output ports (Alpha, Num, Other) according to "attr" components - alphabet only, digits only, other.
If | @Evaluate([string is alpha -strict {@Value(attr)}]) = 1 | Alpha
Else If | @Evaluate([string is digit -strict {@Value(attr)}]) = 1 | Num
Else | <All Other Conditions> | Other
StringConcatenator with this setting creates a concatenated string consisting of initial characters of "first" and "last". When "first" = "Takashi" and "last" = "Iijima", for example, the result will be "TI".
@Evaluate([string index {@Value(first)} 0])@Evaluate([string index {@Value(last)} 0])
* Shown in the Advanced String Editor
FeatureMerger with this setting merges features on the first character of requester's "attr" and the value of supplier's "initial"; assume "initial" contains one character.
Requestor: @Evaluate([string index {@Value(attr)} 0])
Supplier: initial (an attribute name of the supplier)
Not only these transformers but also some other transformers might accept Tcl commands. And some Tcl commands can be replaced with FME String Functions.
When does the Tcl interpreter "compile" those commands?
Here is a thought-provoking description.
"A drawback of the ExpressionEvaluator is that it uses TCL and therefore the TCL interpreter. Every time a feature passes through it, the expression has to be "compiled" before the calculation can be made. So 1 million features = 1 million compilations!"
-- FMEpedia > ExpressionEvaluator > ExpressionEvaluator Performance
Guessing from this, Tcl commands embedded in a transformer parameter probably will be "compiled" every time a feature passed in (am I wrong?). We should be aware of this point especially when we have to process a very large number of features. In certain cases, Python script could be more efficient than Tcl commands.
Of course Python is not always the best solution; Python is just one of the choices. The best solution should be determined according to the actual conditions and several viewpoints - not only efficiency but also maintainability, understandability etc.. I think that the important thing is to have many choices as long as possible, to create more effective workspaces.
Next: Scripted Parameter
=====
2015-02-22 Note: The @Evaluate function of FME 2015 will not return non-numeric value any longer. If the Tcl expression which is embedded to the argument for the function returns non-numeric, the @Evaluate functions will return <null>. Therefore, the following examples are unavailable in FME 2015 unfortunately.
=====
Several transformers other than the TclCaller and the AttributeCreator also accept Tcl commands in the parameter setting. Here I'll give some examples.
FME 2013 SP4 Build 13545 Win32
In the following examples, "string" command is used with various options.
Tcl Commands > string
Tester with this setting decides whether the length (number of characters) of "attr" is less than 10.
@Evaluate([string length {@Value(attr)}]) < 10
TestFilter with this setting routes features into 3 different output ports (Alpha, Num, Other) according to "attr" components - alphabet only, digits only, other.
If | @Evaluate([string is alpha -strict {@Value(attr)}]) = 1 | Alpha
Else If | @Evaluate([string is digit -strict {@Value(attr)}]) = 1 | Num
Else | <All Other Conditions> | Other
StringConcatenator with this setting creates a concatenated string consisting of initial characters of "first" and "last". When "first" = "Takashi" and "last" = "Iijima", for example, the result will be "TI".
@Evaluate([string index {@Value(first)} 0])@Evaluate([string index {@Value(last)} 0])
* Shown in the Advanced String Editor
FeatureMerger with this setting merges features on the first character of requester's "attr" and the value of supplier's "initial"; assume "initial" contains one character.
Requestor: @Evaluate([string index {@Value(attr)} 0])
Supplier: initial (an attribute name of the supplier)
Not only these transformers but also some other transformers might accept Tcl commands. And some Tcl commands can be replaced with FME String Functions.
When does the Tcl interpreter "compile" those commands?
Here is a thought-provoking description.
"A drawback of the ExpressionEvaluator is that it uses TCL and therefore the TCL interpreter. Every time a feature passes through it, the expression has to be "compiled" before the calculation can be made. So 1 million features = 1 million compilations!"
-- FMEpedia > ExpressionEvaluator > ExpressionEvaluator Performance
Guessing from this, Tcl commands embedded in a transformer parameter probably will be "compiled" every time a feature passed in (am I wrong?). We should be aware of this point especially when we have to process a very large number of features. In certain cases, Python script could be more efficient than Tcl commands.
Of course Python is not always the best solution; Python is just one of the choices. The best solution should be determined according to the actual conditions and several viewpoints - not only efficiency but also maintainability, understandability etc.. I think that the important thing is to have many choices as long as possible, to create more effective workspaces.
Next: Scripted Parameter
2013-10-12
FME 2013 SP4 has been released!
A few days ago, SP4 has been released. Of course, I upgraded my FME already.
> FME 2013 SP4: The End of the Line -- Dale Lutz
And, the FME Evangelist has moved. Previous Evangelist articles are in "About FME Archives" category.
> Safe Software Blog > About FME Archives
> FME 2013 SP4: The End of the Line -- Dale Lutz
And, the FME Evangelist has moved. Previous Evangelist articles are in "About FME Archives" category.
> Safe Software Blog > About FME Archives
2013-10-09
FME Objects Python API with External Python (Windows)
FME has its own Python environment, and FME uses it by default when running a workspace; FME Objects Python API (fmeobjects module) can be used in the environment via scripts embedded in an FME workspace, i.e. Startup / Shutdown Script, PythonCreator and PythonCaller.
We can also use FME Objects in an external Python environment. Of course, FME and the Python environment have to be installed in the same machine.
Tested in Windows Xp SP3, FME 2013 SP3 and Python 2.7 >> Download Python
Installing fmeobjects module:
Just copy this Python Dynamic Module file
<FME>\fmeobjects\python27\fmeobjects.pyd
to
<Python27>\Lib\site-packages\fmeobjects.pyd
<FME> is FME home directory, "C:\Program Files\FME" by default.
<Python27> is Python 2.7 home directory, "C:\Python27" by default.
Testing with the external Python (command line):
-----
>>> import fmeobjects
>>> licMan = fmeobjects.FMELicenseManager()
>>> print licMan.getLicenseType()
MACHINE SPECIFIC
>>> wr = fmeobjects.FMEWorkspaceRunner()
>>> wr.run('C:/tmp/test.fmw')
-----
Alternatively, we can also append module searching path before importing fmeobjects module. In this case, we don't need to copy "fmeobjects.pyd" to the external Python environment, but the FME home directory path should be known.
-----
>>> import sys
>>> sys.path.append('<FME>/fmeobjects/python27')
>>> import fmeobjects
-----
That's all. Very simple!
> Python and Available Licenses
We can also use FME Objects in an external Python environment. Of course, FME and the Python environment have to be installed in the same machine.
Tested in Windows Xp SP3, FME 2013 SP3 and Python 2.7 >> Download Python
Just copy this Python Dynamic Module file
<FME>\fmeobjects\python27\fmeobjects.pyd
to
<Python27>\Lib\site-packages\fmeobjects.pyd
<FME> is FME home directory, "C:\Program Files\FME" by default.
<Python27> is Python 2.7 home directory, "C:\Python27" by default.
Testing with the external Python (command line):
-----
>>> import fmeobjects
>>> licMan = fmeobjects.FMELicenseManager()
>>> print licMan.getLicenseType()
MACHINE SPECIFIC
>>> wr = fmeobjects.FMEWorkspaceRunner()
>>> wr.run('C:/tmp/test.fmw')
-----
Alternatively, we can also append module searching path before importing fmeobjects module. In this case, we don't need to copy "fmeobjects.pyd" to the external Python environment, but the FME home directory path should be known.
-----
>>> import sys
>>> sys.path.append('<FME>/fmeobjects/python27')
>>> import fmeobjects
-----
That's all. Very simple!
> Python and Available Licenses
2013-10-08
Getting Started with Tcl in FME: AttributeCreator
Previous: TclCaller
=====
2015-02-22 Note: The @Evaluate function of FME 2015 will not return non-numeric value any longer. If the Tcl expression which is embedded to the argument for the function returns non-numeric, the @Evaluate functions will return <null>. Therefore, the following examples are unavailable in FME 2015 unfortunately.
=====
Tcl commands can be embedded in a "Value" parameter of the AttributeCreator transformer; for example:
@Evaluate([string toupper {@Value(attr_name)}])
- Source: Evaluate Tcl Commands in AttributeCreator
- Destination: EVALUATE TCL COMMANDS IN ATTRIBUTECREATOR
Tcl Commands > string toupper
@Value function puts an existing attribute value specified by its name, @Evaluate function evaluates Tcl commands within brackets and returns the result value.
On the Arithmetic Editor, we don't need to type "@Evaluate" and outer brackets. Write only Tcl commands bracketed by "[ ... ]", it will be bracketed with "@Evaluate(...)" automatically after closing the editor.
Several examples from the Community:
RegEx for finding more than one letter in a string
@Evaluate([regsub -all {[^A-Z]+} {@Value(attr1)} {}])
- Source: Apples Bananas Cherries
- Destination: ABC
Tcl Commands > regsub
Sort only part of an attribute in a list
@Evaluate([regsub {(.+)_(.+)} {@Value(attr2)} {\2_\1}])
- Source: 34N27W_100700
- Destination: 100700_34N27W
Regular Expression evaluation
@Evaluate([join [regexp -all -inline -- {[^0-9]+[0-9]+} {@Value(attr3)}] {,}])
- Source: F29920110716104845F37920120929203346TC20080508184800
- Destination: F29920110716104845,F37920120929203346,TC20080508184800
Tcl Commands > join, regexp
StringReplace to find multiple attributes
@Evaluate([string map {Red Stop Green Go Blue Other} {@Value(attr4)}])
- Source: Green
- Destination: Go
Tcl Commands > string map
Attribute Trimmer
@Evaluate([
array set s [regsub {^(.+)(...)$} {@Value(attr5)} {front {\1} back {\2}}]
return $s(front)[format {%03d} [expr [string trimleft $s(back) {0}] + 1]]
])
- Source: 4VK012
- Destination: 4VK013
TclCommand > array, format, expr, string trimleft, return
Also multiple command lines are acceptable like the example above.
Note: Since Tcl interpreter will try to interpret a string starting with 0 (zero) as an octal number when performing arithmetic processing, we should trim leading zeros if the string have to be interpreted as a decimal number.
Although, of course, these examples can be replaced with existing transformers, the AttributeCreator with Tcl commands could be useful to prevent a cluttered workspace when many string operations have to be performed simultaneously.
Next: Other Transformers
=====
2015-02-22 Note: The @Evaluate function of FME 2015 will not return non-numeric value any longer. If the Tcl expression which is embedded to the argument for the function returns non-numeric, the @Evaluate functions will return <null>. Therefore, the following examples are unavailable in FME 2015 unfortunately.
=====
Tcl commands can be embedded in a "Value" parameter of the AttributeCreator transformer; for example:
@Evaluate([string toupper {@Value(attr_name)}])
- Source: Evaluate Tcl Commands in AttributeCreator
- Destination: EVALUATE TCL COMMANDS IN ATTRIBUTECREATOR
Tcl Commands > string toupper
@Value function puts an existing attribute value specified by its name, @Evaluate function evaluates Tcl commands within brackets and returns the result value.
On the Arithmetic Editor, we don't need to type "@Evaluate" and outer brackets. Write only Tcl commands bracketed by "[ ... ]", it will be bracketed with "@Evaluate(...)" automatically after closing the editor.
Several examples from the Community:
RegEx for finding more than one letter in a string
@Evaluate([regsub -all {[^A-Z]+} {@Value(attr1)} {}])
- Source: Apples Bananas Cherries
- Destination: ABC
Tcl Commands > regsub
Sort only part of an attribute in a list
@Evaluate([regsub {(.+)_(.+)} {@Value(attr2)} {\2_\1}])
- Source: 34N27W_100700
- Destination: 100700_34N27W
Regular Expression evaluation
@Evaluate([join [regexp -all -inline -- {[^0-9]+[0-9]+} {@Value(attr3)}] {,}])
- Source: F29920110716104845F37920120929203346TC20080508184800
- Destination: F29920110716104845,F37920120929203346,TC20080508184800
Tcl Commands > join, regexp
StringReplace to find multiple attributes
@Evaluate([string map {Red Stop Green Go Blue Other} {@Value(attr4)}])
- Source: Green
- Destination: Go
Tcl Commands > string map
Attribute Trimmer
@Evaluate([
array set s [regsub {^(.+)(...)$} {@Value(attr5)} {front {\1} back {\2}}]
return $s(front)[format {%03d} [expr [string trimleft $s(back) {0}] + 1]]
])
- Source: 4VK012
- Destination: 4VK013
TclCommand > array, format, expr, string trimleft, return
Also multiple command lines are acceptable like the example above.
Note: Since Tcl interpreter will try to interpret a string starting with 0 (zero) as an octal number when performing arithmetic processing, we should trim leading zeros if the string have to be interpreted as a decimal number.
Although, of course, these examples can be replaced with existing transformers, the AttributeCreator with Tcl commands could be useful to prevent a cluttered workspace when many string operations have to be performed simultaneously.
Next: Other Transformers
2013-10-05
Getting Started with Tcl in FME: TclCaller
I've never used Tcl so far, but I recently understood its availability in FME especially on string processing. So, I decided to learn it.
I would start from the TclCaller transformer. The subject matter of examples is how to extract only upper case characters from a string attribute. This subject is from:
> RegEx for finding more than one letter in a string
Naturally, such an operation can be performed using the StringReplacer. The followings are just for getting started.
Where can we define Tcl commands for the TclCaller?
There are 3 ways to speficy Tcl commands for the TclCaller.
1. Define command(s) in "Tcl Expression" parameter
2. Define procedure(s) in "Source Code" parameter and use them in "Tcl Expression"
3. Define procedure(s) in an external file, specify the file path to "External Tcl Source File" parameter and use them in "Tcl Expression"
These are also mixable in the same TclCaller. But procedures defined in other TclCaller or Startup / Shutdown TCL Script cannot be used.
Example 1: Define commands in "Tcl Expression"
We can define commands in "Tcl Expression" directly.
FME pre-defined procedures named "FME_***" can be used here.
-----
regsub -all {[^A-Z]+} [FME_GetAttribute attr_name] {}
-----
When the attribute named "attr_name" contains "Apples Bananas Cherries", this command returns "ABC".
Tcl Commands > regsub
Example 2: Define procedures in "Source Code" or external file
In this case, we should define at least one procedure, and use it in "Tcl Expression".
FME pre-defined procedures can be used in any procedure.
-----
proc processFeature {} {
return [regsub -all {[^A-Z]+} [FME_GetAttribute attr_name] {}]
}
-----
Tcl Expression: processFeature
-----
Tcl Commands > proc, return
The next procedure is equivalent with above, because Tcl procedure returns the value returned from the last command if there is not "return" command.
-----
proc processFeature {} {
regsub -all {[^A-Z]+} [FME_GetAttribute attr_name] {}
}
-----
And this setting is also available.
-----
Source Code or External File: proc processFeature {s} {regsub -all {[^A-Z]+} $s {}}
-----
Tcl Expression: processFeature [FME_GetAttribute attr_name]
-----
Destination Attribute
"Tcl Expression" may return a value like the examples above, and the returned value will be saved in an attribute of the output feature. The attribute name will be the string specified as "Destination Attribute" parameter, named "_result" by default.
"Destination Attribute" parameter is not optional. If "Tcl Expression" returns no value, the destination attribute will be empty.
If the input feature had an attribute whose name is same as the destination attribute, the attribute value will be overwritten.
The TclCaller seems to always process the input features one by one. We cannot implement "group-based" processing using the TclCaller, it's a difference from the PythonCaller.
Next: AttributeCreator
I would start from the TclCaller transformer. The subject matter of examples is how to extract only upper case characters from a string attribute. This subject is from:
> RegEx for finding more than one letter in a string
Naturally, such an operation can be performed using the StringReplacer. The followings are just for getting started.
Where can we define Tcl commands for the TclCaller?
There are 3 ways to speficy Tcl commands for the TclCaller.
1. Define command(s) in "Tcl Expression" parameter
2. Define procedure(s) in "Source Code" parameter and use them in "Tcl Expression"
3. Define procedure(s) in an external file, specify the file path to "External Tcl Source File" parameter and use them in "Tcl Expression"
These are also mixable in the same TclCaller. But procedures defined in other TclCaller or Startup / Shutdown TCL Script cannot be used.
Example 1: Define commands in "Tcl Expression"
We can define commands in "Tcl Expression" directly.
FME pre-defined procedures named "FME_***" can be used here.
-----
regsub -all {[^A-Z]+} [FME_GetAttribute attr_name] {}
-----
When the attribute named "attr_name" contains "Apples Bananas Cherries", this command returns "ABC".
Tcl Commands > regsub
Example 2: Define procedures in "Source Code" or external file
In this case, we should define at least one procedure, and use it in "Tcl Expression".
FME pre-defined procedures can be used in any procedure.
-----
proc processFeature {} {
return [regsub -all {[^A-Z]+} [FME_GetAttribute attr_name] {}]
}
-----
Tcl Expression: processFeature
-----
Tcl Commands > proc, return
The next procedure is equivalent with above, because Tcl procedure returns the value returned from the last command if there is not "return" command.
-----
proc processFeature {} {
regsub -all {[^A-Z]+} [FME_GetAttribute attr_name] {}
}
-----
And this setting is also available.
-----
Source Code or External File: proc processFeature {s} {regsub -all {[^A-Z]+} $s {}}
-----
Tcl Expression: processFeature [FME_GetAttribute attr_name]
-----
"Tcl Expression" may return a value like the examples above, and the returned value will be saved in an attribute of the output feature. The attribute name will be the string specified as "Destination Attribute" parameter, named "_result" by default.
"Destination Attribute" parameter is not optional. If "Tcl Expression" returns no value, the destination attribute will be empty.
If the input feature had an attribute whose name is same as the destination attribute, the attribute value will be overwritten.
The TclCaller seems to always process the input features one by one. We cannot implement "group-based" processing using the TclCaller, it's a difference from the PythonCaller.
Next: AttributeCreator