FME Memorandum: September 2013

2013-09-30

FME stores all attributes as character strings: Part 2

I quoted this description from the FME Workbench documentation in the previous post.
> FME Workbench > FME Architecture (Reference) > Attributes
"Feature attributes are usually a primitive type: integers, floats, characters. Internally, FME stores all attributes as character strings and automatically converts between a string representation and a numeric representation as needed."
=====
2015-05-07: Oops, the link is invalid now. Maybe it has been removed...
=====

However, some attributes created by certain transformers seem to be non-string objects. For example, the type of _part_number created by the Deaggregator will be integer. So, I think the description should be understood as a "principle".

As long as processing attributes via existing transformers, this will cause a problem rarely. But we should be aware that there could be different data types when processing attributes in a Python script.

I believe the most important thing is that Python uses two types to represent character strings. i.e. "str" and "unicode".
I think FME users especially with non-English (non-ASCII) locale often encounter errors saying <UnicodeEncodeError> when trying to process character strings in Python script. Such an error seems to occur when a "unicode" instance can not be interpreted to a "str" instance.
To avoid the error, we should encode explicitly the string if it is a unicode instance:
-----
if isinstance(s, unicode):
s = s.encode("<encoding name>")
-----

Type the actual encoding name to <encoding name>, e.g. "cp932" in Japanese Windows standard locale.
More generally, we can also get the appropriate encoding name with locale.getdefaultlocale function.

> PythonCaller: Use logMessageString Problems with Encoding
Revised just a little:
-----
import fmeobjects
import locale

class FeatureProcessor(object):
def __init__(self):
loc = locale.getdefaultlocale()
self.enc = loc[1] # save the default encoding name
self.logger = fmeobjects.FMELogFile()

def input(self, feature):
s = feature.getAttribute('attr')
if isinstance(s, unicode):
s = s.encode(self.enc)
self.logger.logMessageString(str(s))

def close(self):
pass
-----

Although there is a workaround, it's troublesome a little. Hope the functions of FME Objects Python API can interpret always "unicode" instances automatically.

> Unicode HOWTO

2013-09-29

Condense Python Script with List Manipulations

=====
2013-12-23: FME 2014 supports <null>; a list attribute can also contain <null> elements. But FMEFeature.getAttribute and setAttribute methods do not treat <null> elements as <null>.
We should be aware this point in FME 2014+.
See more information here > Null in FME 2014: Handling Null with Python / Tcl
=====

Python API - fmeobjects.FMEFeature.getAttribute, setAttribute and removeAttribute functions treat list attributes (except nested lists) in the same manner as non-list attributes, as long as list elements are string instances. And also Python language has convenient and powerful mechanisms to manipulate list structures.
Therefore, a script with list manipulations could be condensed using those mechanisms effectively.

> Sharing: "Extracting a schema subset for dynamic schemas"
The script I've posted to the thread above, for example, can be replaced with this.
-----
# Example for "Extracting a schema subset for dynamic schemas"
# Last update: 2013-09-30
import fmeobjects

ListNames = ['attribute{}.' + s for s in ['name', 'native_data_type', 'fme_data_type']]

def keep_defs(feature):
attr = map(feature.getAttribute, ListNames)
keeps = [s.strip() for s in str(feature.getAttribute('_to_keep')).split(',')]
if attr[0]:
# t will be a tuple consisting of name, native_data_type and fme_data_type,
# "name" matches with one of "keeps".
newAttr = map(list, zip(*[t for t in zip(*attr) if t[0] in keeps]))
map(feature.removeAttribute, ListNames)
map(feature.setAttribute, ListNames, newAttr)
-----

This script is also possible, but probably is over-condensed one...
-----
import fmeobjects

ListNames = ['attribute{}.' + s for s in ['name', 'native_data_type', 'fme_data_type']]

def keep_defs(feature):
attr = map(feature.getAttribute, ListNames)
keeps = [s.strip() for s in str(feature.getAttribute('_to_keep')).split(',')]
if attr[0]:
map(lambda n, v: (feature.removeAttribute(n), feature.setAttribute(n, v)),
ListNames, map(list, zip(*[t for t in zip(*attr) if t[0] in keeps])))
-----

=====
2014-01-26: More than enough is too much. Concise script is not always good.
See also "Efficiency of List Attribute Manipulation with Python".
=====

References:
List Comprehensions
Built -in "map" Function, "zip" Function
Unpacking Argument Lists
Lambda Expressions

2013-09-25

Create FME Parameter Dialog Box with Python Script

I posted a way to create a message dialog box in the Community site.
> Message or Alert Box
That was only to show a message string, I think it's a special usage of FME parameter dialog box. Here I put an example for general usage. FME GUI Directives and file operations are the points.
-----
import fmeobjects
import os, tempfile, winsound

# FME GUI directives
kGuiDirectives = """\
GUI TITLE Context Parameters
GUI GROUP 1%PARAM1%PARAM2%PARAM3 Context Parameters
GUI TEXT PARAM1 Text Parameter:
GUI INTEGER PARAM2 Integer Parameter:
GUI FLOAT PARAM3 Float Parameter:
"""

class FeatureProcessor(object):
def __init__(self):
# Dictionary to save parameter values.
self.params = {}

# Flag indicates whether parameters are entered or not.
self.complete = False

# If it's not necessary to enter parameters in a certain condition,
# test the condition and finish this function when it's true.
# if <condition>:
# self.complete = True
# return

filepath = None
try:
# Create a temporary file defining FME GUI directives.
fd, filepath = tempfile.mkstemp(dir = '.')
os.write(fd, kGuiDirectives)
os.close(fd)

# *** Windows only ***
# Play a Windows sound before showing the dialog box.
winsound.PlaySound('SystemAsterisk', winsound.SND_ASYNC)

# Create and show a parameter settings dialog box.
# If [OK] button is clicked to close the dialog box,
# save entered parameter values.
# FME Objects Python API description:
# "If the user exits the form with OK, then the file will be overwritten
# with a new file that alternates MACRO NAME and VALUE, line by line."
dlg = fmeobjects.FMEDialog()
if dlg.parameterPrompt(filepath):
f = open(filepath)
rows = [r.strip() for r in f.readlines()]
self.params = dict(zip(rows[0::2], rows[1::2]))
f.close()
self.complete = True
except:
logger = fmeobjects.FMELogFile()
logger.logMessageString('Parameters getting failure.',
fmeobjects.FME_ERROR)
finally:
# Remove the temporary file.
if filepath and os.path.exists(filepath):
os.remove(filepath)

def input(self, feature):
# If parameters have not been entered, abort the translation.
if not self.complete:
raise fmeobjects.FMEException('Parameters have not been entered.')

# Use the parameter values.
# Set them to each input feature, for example.
for name in self.params.keys():
feature.setAttribute(name, self.params[name])
self.pyoutput(feature)

def close(self):
pass
-----
=====
2014-11-29: An application.
Community Answers > select different feature type each time workspace runs
=====

Related links:
> FME GUI Directives * currently disabled. 2013-10-28
> FME GUI DIRECTIVES
> 35.4. winsound — Sound-playing interface for Windows

WorkspaceRunner Examples

In past, there was an article describing WorkspaceRunnder examples in FMEpedia. Although there is a link to the URL in the WorkspaceRunner transformer description (Workbench help), the linked page is currently disabled. Where has it gone?
-----
2013-10-02
The article has been updated, 1 Oct. 2013. Link from Workbench help is also available.
Thank you, Safe support team!
FMEpedia: Batch Processing Using the WorkspaceRunner
-----

Usage of the WorkspaceRunner is not so difficult. But many beginners seem to feel it's difficult to understand how it works, as I was so. In fact, questions about the WorksapceRunner are often posted in Community, I recently posted a couple of examples to correspond to such a question.

> Creating a BATCH with multiple INPUTS (variables) to run a workbench that intersects LINES AND POLYGONS
> Workspace Runner and Path Format Reader example
For these questions I provided example workspaces showing the most basic usage of the WorkspaceRunner - i.e. performing simple batch processing.
Download (3MB zipped file including test data)

> Dynamic schema partial merge from 2 data sources
Although the subject of this question is an application of Dynamic Schema functionality, I thought it can be realized using the WorkspaceRunners, and created applicable workspaces. It was a very interesting challenge.
Download (32KB zipped file, including test data)
-----
2013-09-27
However, a custom format could be more elegant solution in this case.
Download: Custom Format Example 0 (22KB) using Python script
Download: Custom Format Example 1 (23KB) non-scripted version
Related article: FMEpedia > Extracting a schema subset for dynamic schemas
-----

Through creating these examples, I felt that knowing well about Published Parameters is the key to understand how the WorkspaceRunner works.
~~Anyway, I hope that the upgraded article is uploaded to FMEpedia in the near future.~~

2013-09-11

What does the InsidePointReplacer create?

FME provides three transformers to create a point related to an area feature.
CenterOfGravityReplacer creates the center of mass,
CenterPointReplacer creates the center of the bounding box,
InsidePointReplacer creates an inside point of the original area.

The point created by the InsidePointReplacer is guaranteed to be inside of the original area, but other transformers may create an outside point. The center point or the center of mass of U or L shape probably is outside, for example.
If knowing what kind of point will be created, there are no practical problems.

However, I want to know the actual algorithm that is used in the InsidePointReplacer.
If the original shape is a triangle or a rectangle, the inside point seems to match with the center of mass. Otherwise, the inside point is different from the center of mass even if that is inside the area.
The algorithm is in a black box.

> Community: Deaggregating polygons andresultant centroid position

2013-09-01

Extract List Attribute Elements in Python Script

To extract list attribute elements and create non-list attributes storing the values:
Simple List: _list{i} --> attr_i
# Example 1
def extractListElements(feature):
i = 0
while True:
v = feature.getAttribute('_list{%d}' % i)
if v == None:
break
feature.setAttribute('attr_%d' % i, v)
i += 1

# Example 2
def extractListElements(feature):
list = feature.getAttribute('_list{}')
if list != None:
for i, v in enumerate(list):
feature.setAttribute('attr_%d' % i, v)

Complex List: _list{i}.attr --> attr_i
# Example 3
def extractComplexListElements(feature):
i = 0
while True:
v = feature.getAttribute('_list{%d}.attr' % i)
if v == None:
break
feature.setAttribute('attr_%d' % i, v)
i += 1

# Example 4
def extractComplexListElements(feature):
list = feature.getAttribute('_list{}.attr')
if list != None:
for i, v in enumerate(list):
feature.setAttribute('attr_%d' % i, v)

Nested List: _list{i}.sub{j} --> attr_ij
# Example 5
def extractNestedListElements(feature):
i = 0
while True:
sub = feature.getAttribute('_list{%d}.sub{}' % i)
if sub == None:
break
for j, v in enumerate(sub):
feature.setAttribute('attr_%d%d' % (i, j), v)
i += 1

Related articles:
Community > Set a list attribute in Python
Community > FME Terminologies about List Attribute