A few days ago, SP4 has been released. Of course, I upgraded my FME already.
> FME 2013 SP4: The End of the Line -- Dale Lutz
And, the FME Evangelist has moved. Previous Evangelist articles are in "About FME Archives" category.
> Safe Software Blog > About FME Archives
2013-10-12
2013-10-09
FME Objects Python API with External Python (Windows)
FME has its own Python environment, and FME uses it by default when running a workspace; FME Objects Python API (fmeobjects module) can be used in the environment via scripts embedded in an FME workspace, i.e. Startup / Shutdown Script, PythonCreator and PythonCaller.
We can also use FME Objects in an external Python environment. Of course, FME and the Python environment have to be installed in the same machine.
Tested in Windows Xp SP3, FME 2013 SP3 and Python 2.7 >> Download Python
Installing fmeobjects module:
Just copy this Python Dynamic Module file
<FME>\fmeobjects\python27\fmeobjects.pyd
to
<Python27>\Lib\site-packages\fmeobjects.pyd
<FME> is FME home directory, "C:\Program Files\FME" by default.
<Python27> is Python 2.7 home directory, "C:\Python27" by default.
Testing with the external Python (command line):
-----
>>> import fmeobjects
>>> licMan = fmeobjects.FMELicenseManager()
>>> print licMan.getLicenseType()
MACHINE SPECIFIC
>>> wr = fmeobjects.FMEWorkspaceRunner()
>>> wr.run('C:/tmp/test.fmw')
-----
Alternatively, we can also append module searching path before importing fmeobjects module. In this case, we don't need to copy "fmeobjects.pyd" to the external Python environment, but the FME home directory path should be known.
-----
>>> import sys
>>> sys.path.append('<FME>/fmeobjects/python27')
>>> import fmeobjects
-----
That's all. Very simple!
> Python and Available Licenses
We can also use FME Objects in an external Python environment. Of course, FME and the Python environment have to be installed in the same machine.
Tested in Windows Xp SP3, FME 2013 SP3 and Python 2.7 >> Download Python
Just copy this Python Dynamic Module file
<FME>\fmeobjects\python27\fmeobjects.pyd
to
<Python27>\Lib\site-packages\fmeobjects.pyd
<FME> is FME home directory, "C:\Program Files\FME" by default.
<Python27> is Python 2.7 home directory, "C:\Python27" by default.
Testing with the external Python (command line):
-----
>>> import fmeobjects
>>> licMan = fmeobjects.FMELicenseManager()
>>> print licMan.getLicenseType()
MACHINE SPECIFIC
>>> wr = fmeobjects.FMEWorkspaceRunner()
>>> wr.run('C:/tmp/test.fmw')
-----
Alternatively, we can also append module searching path before importing fmeobjects module. In this case, we don't need to copy "fmeobjects.pyd" to the external Python environment, but the FME home directory path should be known.
-----
>>> import sys
>>> sys.path.append('<FME>/fmeobjects/python27')
>>> import fmeobjects
-----
That's all. Very simple!
> Python and Available Licenses
2013-10-08
Getting Started with Tcl in FME: AttributeCreator
Previous: TclCaller
=====
2015-02-22 Note: The @Evaluate function of FME 2015 will not return non-numeric value any longer. If the Tcl expression which is embedded to the argument for the function returns non-numeric, the @Evaluate functions will return <null>. Therefore, the following examples are unavailable in FME 2015 unfortunately.
=====
Tcl commands can be embedded in a "Value" parameter of the AttributeCreator transformer; for example:
@Evaluate([string toupper {@Value(attr_name)}])
- Source: Evaluate Tcl Commands in AttributeCreator
- Destination: EVALUATE TCL COMMANDS IN ATTRIBUTECREATOR
Tcl Commands > string toupper
@Value function puts an existing attribute value specified by its name, @Evaluate function evaluates Tcl commands within brackets and returns the result value.
On the Arithmetic Editor, we don't need to type "@Evaluate" and outer brackets. Write only Tcl commands bracketed by "[ ... ]", it will be bracketed with "@Evaluate(...)" automatically after closing the editor.
Several examples from the Community:
RegEx for finding more than one letter in a string
@Evaluate([regsub -all {[^A-Z]+} {@Value(attr1)} {}])
- Source: Apples Bananas Cherries
- Destination: ABC
Tcl Commands > regsub
Sort only part of an attribute in a list
@Evaluate([regsub {(.+)_(.+)} {@Value(attr2)} {\2_\1}])
- Source: 34N27W_100700
- Destination: 100700_34N27W
Regular Expression evaluation
@Evaluate([join [regexp -all -inline -- {[^0-9]+[0-9]+} {@Value(attr3)}] {,}])
- Source: F29920110716104845F37920120929203346TC20080508184800
- Destination: F29920110716104845,F37920120929203346,TC20080508184800
Tcl Commands > join, regexp
StringReplace to find multiple attributes
@Evaluate([string map {Red Stop Green Go Blue Other} {@Value(attr4)}])
- Source: Green
- Destination: Go
Tcl Commands > string map
Attribute Trimmer
@Evaluate([
array set s [regsub {^(.+)(...)$} {@Value(attr5)} {front {\1} back {\2}}]
return $s(front)[format {%03d} [expr [string trimleft $s(back) {0}] + 1]]
])
- Source: 4VK012
- Destination: 4VK013
TclCommand > array, format, expr, string trimleft, return
Also multiple command lines are acceptable like the example above.
Note: Since Tcl interpreter will try to interpret a string starting with 0 (zero) as an octal number when performing arithmetic processing, we should trim leading zeros if the string have to be interpreted as a decimal number.
Although, of course, these examples can be replaced with existing transformers, the AttributeCreator with Tcl commands could be useful to prevent a cluttered workspace when many string operations have to be performed simultaneously.
Next: Other Transformers
=====
2015-02-22 Note: The @Evaluate function of FME 2015 will not return non-numeric value any longer. If the Tcl expression which is embedded to the argument for the function returns non-numeric, the @Evaluate functions will return <null>. Therefore, the following examples are unavailable in FME 2015 unfortunately.
=====
Tcl commands can be embedded in a "Value" parameter of the AttributeCreator transformer; for example:
@Evaluate([string toupper {@Value(attr_name)}])
- Source: Evaluate Tcl Commands in AttributeCreator
- Destination: EVALUATE TCL COMMANDS IN ATTRIBUTECREATOR
Tcl Commands > string toupper
@Value function puts an existing attribute value specified by its name, @Evaluate function evaluates Tcl commands within brackets and returns the result value.
On the Arithmetic Editor, we don't need to type "@Evaluate" and outer brackets. Write only Tcl commands bracketed by "[ ... ]", it will be bracketed with "@Evaluate(...)" automatically after closing the editor.
Several examples from the Community:
RegEx for finding more than one letter in a string
@Evaluate([regsub -all {[^A-Z]+} {@Value(attr1)} {}])
- Source: Apples Bananas Cherries
- Destination: ABC
Tcl Commands > regsub
Sort only part of an attribute in a list
@Evaluate([regsub {(.+)_(.+)} {@Value(attr2)} {\2_\1}])
- Source: 34N27W_100700
- Destination: 100700_34N27W
Regular Expression evaluation
@Evaluate([join [regexp -all -inline -- {[^0-9]+[0-9]+} {@Value(attr3)}] {,}])
- Source: F29920110716104845F37920120929203346TC20080508184800
- Destination: F29920110716104845,F37920120929203346,TC20080508184800
Tcl Commands > join, regexp
StringReplace to find multiple attributes
@Evaluate([string map {Red Stop Green Go Blue Other} {@Value(attr4)}])
- Source: Green
- Destination: Go
Tcl Commands > string map
Attribute Trimmer
@Evaluate([
array set s [regsub {^(.+)(...)$} {@Value(attr5)} {front {\1} back {\2}}]
return $s(front)[format {%03d} [expr [string trimleft $s(back) {0}] + 1]]
])
- Source: 4VK012
- Destination: 4VK013
TclCommand > array, format, expr, string trimleft, return
Also multiple command lines are acceptable like the example above.
Note: Since Tcl interpreter will try to interpret a string starting with 0 (zero) as an octal number when performing arithmetic processing, we should trim leading zeros if the string have to be interpreted as a decimal number.
Although, of course, these examples can be replaced with existing transformers, the AttributeCreator with Tcl commands could be useful to prevent a cluttered workspace when many string operations have to be performed simultaneously.
Next: Other Transformers
2013-10-05
Getting Started with Tcl in FME: TclCaller
I've never used Tcl so far, but I recently understood its availability in FME especially on string processing. So, I decided to learn it.
I would start from the TclCaller transformer. The subject matter of examples is how to extract only upper case characters from a string attribute. This subject is from:
> RegEx for finding more than one letter in a string
Naturally, such an operation can be performed using the StringReplacer. The followings are just for getting started.
Where can we define Tcl commands for the TclCaller?
There are 3 ways to speficy Tcl commands for the TclCaller.
1. Define command(s) in "Tcl Expression" parameter
2. Define procedure(s) in "Source Code" parameter and use them in "Tcl Expression"
3. Define procedure(s) in an external file, specify the file path to "External Tcl Source File" parameter and use them in "Tcl Expression"
These are also mixable in the same TclCaller. But procedures defined in other TclCaller or Startup / Shutdown TCL Script cannot be used.
Example 1: Define commands in "Tcl Expression"
We can define commands in "Tcl Expression" directly.
FME pre-defined procedures named "FME_***" can be used here.
-----
regsub -all {[^A-Z]+} [FME_GetAttribute attr_name] {}
-----
When the attribute named "attr_name" contains "Apples Bananas Cherries", this command returns "ABC".
Tcl Commands > regsub
Example 2: Define procedures in "Source Code" or external file
In this case, we should define at least one procedure, and use it in "Tcl Expression".
FME pre-defined procedures can be used in any procedure.
-----
proc processFeature {} {
return [regsub -all {[^A-Z]+} [FME_GetAttribute attr_name] {}]
}
-----
Tcl Expression: processFeature
-----
Tcl Commands > proc, return
The next procedure is equivalent with above, because Tcl procedure returns the value returned from the last command if there is not "return" command.
-----
proc processFeature {} {
regsub -all {[^A-Z]+} [FME_GetAttribute attr_name] {}
}
-----
And this setting is also available.
-----
Source Code or External File: proc processFeature {s} {regsub -all {[^A-Z]+} $s {}}
-----
Tcl Expression: processFeature [FME_GetAttribute attr_name]
-----
Destination Attribute
"Tcl Expression" may return a value like the examples above, and the returned value will be saved in an attribute of the output feature. The attribute name will be the string specified as "Destination Attribute" parameter, named "_result" by default.
"Destination Attribute" parameter is not optional. If "Tcl Expression" returns no value, the destination attribute will be empty.
If the input feature had an attribute whose name is same as the destination attribute, the attribute value will be overwritten.
The TclCaller seems to always process the input features one by one. We cannot implement "group-based" processing using the TclCaller, it's a difference from the PythonCaller.
Next: AttributeCreator
I would start from the TclCaller transformer. The subject matter of examples is how to extract only upper case characters from a string attribute. This subject is from:
> RegEx for finding more than one letter in a string
Naturally, such an operation can be performed using the StringReplacer. The followings are just for getting started.
Where can we define Tcl commands for the TclCaller?
There are 3 ways to speficy Tcl commands for the TclCaller.
1. Define command(s) in "Tcl Expression" parameter
2. Define procedure(s) in "Source Code" parameter and use them in "Tcl Expression"
3. Define procedure(s) in an external file, specify the file path to "External Tcl Source File" parameter and use them in "Tcl Expression"
These are also mixable in the same TclCaller. But procedures defined in other TclCaller or Startup / Shutdown TCL Script cannot be used.
Example 1: Define commands in "Tcl Expression"
We can define commands in "Tcl Expression" directly.
FME pre-defined procedures named "FME_***" can be used here.
-----
regsub -all {[^A-Z]+} [FME_GetAttribute attr_name] {}
-----
When the attribute named "attr_name" contains "Apples Bananas Cherries", this command returns "ABC".
Tcl Commands > regsub
Example 2: Define procedures in "Source Code" or external file
In this case, we should define at least one procedure, and use it in "Tcl Expression".
FME pre-defined procedures can be used in any procedure.
-----
proc processFeature {} {
return [regsub -all {[^A-Z]+} [FME_GetAttribute attr_name] {}]
}
-----
Tcl Expression: processFeature
-----
Tcl Commands > proc, return
The next procedure is equivalent with above, because Tcl procedure returns the value returned from the last command if there is not "return" command.
-----
proc processFeature {} {
regsub -all {[^A-Z]+} [FME_GetAttribute attr_name] {}
}
-----
And this setting is also available.
-----
Source Code or External File: proc processFeature {s} {regsub -all {[^A-Z]+} $s {}}
-----
Tcl Expression: processFeature [FME_GetAttribute attr_name]
-----
"Tcl Expression" may return a value like the examples above, and the returned value will be saved in an attribute of the output feature. The attribute name will be the string specified as "Destination Attribute" parameter, named "_result" by default.
"Destination Attribute" parameter is not optional. If "Tcl Expression" returns no value, the destination attribute will be empty.
If the input feature had an attribute whose name is same as the destination attribute, the attribute value will be overwritten.
The TclCaller seems to always process the input features one by one. We cannot implement "group-based" processing using the TclCaller, it's a difference from the PythonCaller.
Next: AttributeCreator
2013-09-30
FME stores all attributes as character strings: Part 2
I quoted this description from the FME Workbench documentation in the previous post.
> FME Workbench > FME Architecture (Reference) > Attributes
"Feature attributes are usually a primitive type: integers, floats, characters. Internally, FME stores all attributes as character strings and automatically converts between a string representation and a numeric representation as needed."
=====
2015-05-07: Oops, the link is invalid now. Maybe it has been removed...
=====
However, some attributes created by certain transformers seem to be non-string objects. For example, the type of _part_number created by the Deaggregator will be integer. So, I think the description should be understood as a "principle".
As long as processing attributes via existing transformers, this will cause a problem rarely. But we should be aware that there could be different data types when processing attributes in a Python script.
I believe the most important thing is that Python uses two types to represent character strings. i.e. "str" and "unicode".
I think FME users especially with non-English (non-ASCII) locale often encounter errors saying <UnicodeEncodeError> when trying to process character strings in Python script. Such an error seems to occur when a "unicode" instance can not be interpreted to a "str" instance.
To avoid the error, we should encode explicitly the string if it is a unicode instance:
-----
if isinstance(s, unicode):
s = s.encode("<encoding name>")
-----
Type the actual encoding name to <encoding name>, e.g. "cp932" in Japanese Windows standard locale.
More generally, we can also get the appropriate encoding name with locale.getdefaultlocale function.
> PythonCaller: Use logMessageString Problems with Encoding
Revised just a little:
-----
import fmeobjects
import locale
class FeatureProcessor(object):
def __init__(self):
loc = locale.getdefaultlocale()
self.enc = loc[1] # save the default encoding name
self.logger = fmeobjects.FMELogFile()
def input(self, feature):
s = feature.getAttribute('attr')
if isinstance(s, unicode):
s = s.encode(self.enc)
self.logger.logMessageString(str(s))
def close(self):
pass
-----
Although there is a workaround, it's troublesome a little. Hope the functions of FME Objects Python API can interpret always "unicode" instances automatically.
> Unicode HOWTO
> FME Workbench > FME Architecture (Reference) > Attributes
"Feature attributes are usually a primitive type: integers, floats, characters. Internally, FME stores all attributes as character strings and automatically converts between a string representation and a numeric representation as needed."
=====
2015-05-07: Oops, the link is invalid now. Maybe it has been removed...
=====
However, some attributes created by certain transformers seem to be non-string objects. For example, the type of _part_number created by the Deaggregator will be integer. So, I think the description should be understood as a "principle".
As long as processing attributes via existing transformers, this will cause a problem rarely. But we should be aware that there could be different data types when processing attributes in a Python script.
I believe the most important thing is that Python uses two types to represent character strings. i.e. "str" and "unicode".
I think FME users especially with non-English (non-ASCII) locale often encounter errors saying <UnicodeEncodeError> when trying to process character strings in Python script. Such an error seems to occur when a "unicode" instance can not be interpreted to a "str" instance.
To avoid the error, we should encode explicitly the string if it is a unicode instance:
-----
if isinstance(s, unicode):
s = s.encode("<encoding name>")
-----
Type the actual encoding name to <encoding name>, e.g. "cp932" in Japanese Windows standard locale.
More generally, we can also get the appropriate encoding name with locale.getdefaultlocale function.
> PythonCaller: Use logMessageString Problems with Encoding
Revised just a little:
-----
import fmeobjects
import locale
class FeatureProcessor(object):
def __init__(self):
loc = locale.getdefaultlocale()
self.enc = loc[1] # save the default encoding name
self.logger = fmeobjects.FMELogFile()
def input(self, feature):
s = feature.getAttribute('attr')
if isinstance(s, unicode):
s = s.encode(self.enc)
self.logger.logMessageString(str(s))
def close(self):
pass
-----
Although there is a workaround, it's troublesome a little. Hope the functions of FME Objects Python API can interpret always "unicode" instances automatically.
> Unicode HOWTO
Subscribe to:
Posts (Atom)