Efficiency of List Attribute Manipulation with Python

Important
2014-01-29: I heard that Safe is planning to change the implementation of Python API fmeobjects.FMEFeature.getAttribute() method, so that it returns an empty string when specified attribute stores <null>. Currently - FME 2014 build 14234 - it returns "None" in that case.
After confirming the change, I will revise related descriptions (underlined) in this article.
-----
2014-02-14: I noticed that the method in FME 2014 SP1 Beta (build 14255) returns an empty string for <null>. The change of implementation seems to be done for SP1.
-----
2014-02-25: The change about FME Objects Python API has been announced. I revised related descriptions in this article (underlined).

(FME 2014 build 14234)
=====
2015-12-23: I tested again the same scripts shown in this article with FME 2015.1.3.1 build 15573. Contrary to my expectations, the "copyList1" was always faster than others in both Result1 and Result2. The order of others was "copyList2" < "copyList3" < "copyList4" in almost all runs but not always.
=====

There are several Python scripting ways to get or set values of list attribute elements. I tested difference in processing time efficiency among them.
Note: This article describes just an experiment result in my environment (FME 2014 32-bit build 14234, Windows 7 64-bit). The result may be different in other environments and / or conditions.

At first, create a feature which has a list attribute with this script in a PythonCreator. The number of elements is 100000 and every element value is a character string.
-----
import fmeobjects
class FeatureCreator(object):
    def __init__(self):
        pass
     
    def close(self):
        feature = fmeobjects.FMEFeature()
        for i in range(100000):
            feature.setAttribute('_src{%d}' % i, 'string value')
        self.pyoutput(feature)
-----

Then, I measured processing time of these 4 scripts in PythonCaller. All of them copies every element of "_src{}" to "_dest{}". Assume the source list has no <missing> element.
-----
import fmeobjects, time
def copyList1(feature):
    s = time.clock()
    feature.setAttribute('_dest{}', feature.getAttribute('_src{}'))
    feature.setAttribute('_time_diff', '%.3f' % (time.clock() - s))
-----
import fmeobjects, time
def copyList2(feature):
    s = time.clock()
    for i, v in enumerate(feature.getAttribute('_src{}')):
        feature.setAttribute('_dest{%d}' % i, v)
    feature.setAttribute('_time_diff', '%.3f' % (time.clock() - s))
-----
import fmeobjects, time
def copyList3(feature):
    s = time.clock()
    i = 0
    while True:
        value = feature.getAttribute('_src{%d}' % i)
        if value == None:
            break
        feature.setAttribute('_dest{%d}' % i, value)
        i += 1
    feature.setAttribute('_time_diff', '%.3f' % (time.clock() - s))
-----
import fmeobjects, time
def copyList4(feature):
    s = time.clock()
    i = 0
    while True:
        isNull, isMissing, type = feature.getAttributeNullMissingAndType('_src{%d}' % i)
        if isMissing:
            break
        elif isNull:
            feature.setAttributeNullWithType('_dest{%d}' % i, type)
        else:
            feature.setAttribute('_dest{%d}' % i, feature.getAttribute('_src{%d}' % i))
        i += 1
    feature.setAttribute('_time_diff', '%.3f' % (time.clock() - s))
-----
FME 2014 SP1+ (build 14252 or later):
Note: If there were <null> elements in the source list, copyList1, copyList2 and copyList3 will copy <null> elements as empty strings.
FME 2014 without SP*:
Note: If there were <null> elements in the source list, copyList1 and copyList2 will copy <null> elements as empty strings; copyList3 stops copying when it got the first <null> element.
See also "Null in FME 2014: Handling Null with Python / Tcl".

Result 1:  faster <== copyList3 < copyList2 < copyList4 < copyList1 ==> slower

Next, I tested the case where every element is numeric value (really numeric, is not a string which represents number). The source list was created by this script in the PythonCreator.
-----
import fmeobjects
class FeatureCreator(object):
    def __init__(self):
        pass
     
    def close(self):
        feature = fmeobjects.FMEFeature()
        for i in range(100000):
            feature.setAttribute('_src{%d}' % i, 100)
        self.pyoutput(feature)
-----

Result 2: faster <== copyList3 < copyList4 < copyList2 < copyList1 ==> slower

As long as seeing the result, "copyList3" looks always faster than others.
In FME 2014 (without SP*), be aware that "copyList3" cannot be used if the source list can contain <null> elements.

-----
When writing Python script in FME 2014 (without SP*), we always have to be conscious of <null>. Maybe it's troublesome a little. It might be better to convert <null> to a certain value (e.g. empty string etc.) before processing with Python, if possible. I think the NullAttributeMapper is effective to do that.
In FME 2014 SP1+ (build 14252 or later), be aware that FMEFeature.getAttribute method returns an empty string for <null>. When it's necessary to distinguish <null> from an empty string in Python script, consider using FMEFeature.getAttributeNullMissingAndType method.

NullAttributeMapper Use Case: Assign Value to Missing Attributes

(FME 2014 build 14234)

From this thread. > Community: Merge and Flag

Assume there are these two tables.
FileA:

Name	Postcode	DOB
Bob	NG1 1AA	01/01/1970
Sue	WP1 1AA	01/01/1990

FileB:

Name	Postcode	DOB
Bob	NG1 1AA	01/01/1970
John	SW1 1AA	01/01/1980

The question was how to merge the tables; the result table must not contain duplicated person's row, and must have additional columns which indicate existence in the original tables.
Required result table should be:

Name	Postcode	DOB	FileAFlag	FileBFlag
Bob	NG1 1AA	01/01/1970	Yes	Yes
Sue	WP1 1AA	01/01/1990	Yes	No
John	SW1 1AA	01/01/1980	No	Yes

In the Community, an excellent solution using the FeatureMerger has been provided. If I created a workspace according to the solution, the first step for merging tables would be like this.

Immediately after the tables were merged, every "FileBFlag" of NotMerged features and every "FileAFlag" of Unreferenced features are <missing>, i.e. not exist. To satisfy the requirement, finally those <missing> attributes should have a specific value - "No".
The subject I think of here is how to assign "No" to <missing> attributes after merging.

One possible way is to use an AttributeCreator with conditional value setting. "Attribute Is Missing" operator can be used in the setting.

There is another way in FME 2014. That is to use a NullAttributeMapper in order to assign "No" to <missing> FileAFlag and FileBFlag.

Both of them work fine, I cannot say which way is better. But the NullAttributeMapper parameter setting looks simpler than conditional value settings in the AttributeCreator.
I would use the NullAttributeMapper if I encountered a similar requirement.

-----
It's also possible that adding "FileBFlag" (= "No") to NotMatched output and adding "FileAFlag" (= "No") to Unreferenced output in the FeatureMerger using the "Edit Attribute" tool.
If did so, both the AttributeCreator and the NullAttributeMapper would not be necessary no longer, but the process would become difficult to be visible. It's good or not ...

Edit Attribute Tool for Transformers

(FME 2014 build 14232)

I found a nice update in FME 2014. That is "Edit Attribute" tool for Transformers.
This article is describing about the tool for Writers.
> FME 2014 Sneak Peek: Top 10 Workbench Optimizations > 5 Edit Attribute Value

But this tool can be also used for every Transformer.
=====
2014-02-07: Related documentation > Inserting Attributes to Transformer Outputs
=====
First of all, let's confirm when new attributes created by this tool will be added to the input feature. Before processing or After processing of the transformer? (pre-process or post-process?)
By the analogy from the previous functionality i.e. "Add Attribute" + "Insert Constant",  it's expected that new attributes would be added After processing.

Result:

OK, indeed new attributes have been added After processing of the transformer.

The followings are practical use cases which I thought of immediately. Of course all of them can be replaced with existing transformers, but using the "Edit Attribute" tool can make the workspace more concise.

Calculate Width and Height of Bounding Box in a BoundsExtractor
-- surrogate for an AttributeCreator or two ExpressionEvaluator etc.

Extract First Element of List Attribute in an AttributeSplitter
-- surrogate for a ListIndexer etc.

Calculate Length of Line Geometry in a VertexCreator
-- surrogate for a LengthCalculator etc.

Interesting. The "Edit Attribute" tool can be used as something like Expanding Transformer Functionality. There should be many effective use cases.
=====
The "Edit Attribute" tool looks like a pretty baby of the AttributeCreator. I love it :-)

The FMEUserConferencer

The FMEUserConferencer is one of my most favorite Transformers.
I'm planning to join the "transformer". You?

Range-based Attribute Mapping with InlineQuerier

(FME 2014 build 14232)

I remembered by chance that an issue on range-based attribute mapping had been discussed in this thread. > Community: Schema mapper attribute range

Of course the AttributeRangeMapper transformer could be used if the configuration of range-value mapping was static. However, the subject is a case that the range-value mapping will be configured dynamically via an external table at run-time. It's unknown when creating a workspace.
I provided a solution using Python script at that time, but I noticed that the InlineQuerier could be another solution.

Assume "Data" table contains attributes of target features as followings.

ID	Height
1	-5
2	5
3	15
4	25
5	35

And "RangeToLevel" table defines mapping rule of height range and value. For example:

MinHeight	MaxHeight	NewValue
0	10	Level1
10	20	Level2
20	30	Level3

Then, merge those tables using an InlineQuerier.

This is the SQL statement specified to the InlineQuerier. The point is "cross join".
-----
select s.ID, s.Height, t.NewValue
from Data as s left outer join (select a.ID, b.NewValue
    from Data as a cross join RangeToLevel as b
    where b.MinHeight <= a.Height and a.Height < b.MaxHeight) as t on t.ID = s.ID
-----
Result Table:

Here, <null>s have appeared :-)
Yes, the NullAttributeMapper transformer can be used to map every <null> to a preferable value (e.g. "Undefined Level") if necessary.
I think this is also a typical use case of the NullAttributeMapper.

But this SQL statement is also possible to do that. There is always more than one way!
-----
select s.ID, s.Height,
case
    when t.NewValue is null then 'Undefined Level'
    else t.NewValue
end as NewValue
from Data as s left outer join (select a.ID, b.NewValue
    from Data as a cross join RangeToLevel as b
    where b.MinHeight <= a.Height and a.Height < b.MaxHeight) as t on t.ID = s.ID
=====
2014-01-19: If features whose height is out of range can be discarded, or if it's guaranteed that there aren't "out of range" features, this SQL statement can be used simply.
-----
select a.ID, a.Height, b.NewValue
from Data as a cross join RangeToLevel as b
where b.MinHeight <= a.Height and a.Height < b.MaxHeight
-----