2014-03-26

Create World Files to Arrange Images in Tiles

(FME 2014 build 14235)

Imagine that there are hundreds of JPEG files stored in a specific folder.
The goal is to create world files (*.wld) so that the images will be arranged in tile shapes when displaying them on a map viewer (e.g. ArcMap).
Width and height of a tile, and number of columns are specified as constants. Aspect ratio of every image is approximately equal to the ratio of a tile, but their sizes are various.

Required result (simplified example: number of images = 9, number of columns = 3)










A world file is a plain text file in which these 6 parameter values have been written.
Line 1: pixel size in the x-direction in map units/pixel
Line 2: rotation about y-axis
Line 3: rotation about x-axis
Line 4: pixel size in the y-direction in map units, almost always negative
Line 5: x-coordinate of the center of the upper left pixel
Line 6: y-coordinate of the center of the upper left pixel
-- World file, Wikipedia

Line 2 and 3 can be always 0 since it's not necessary to rotate images in this case. I have to calculate other values (Line 1, 4, 5, 6), and write the lines into text files for every image.
Except extension, path of a world file has to be same as associated JPEG file.

OK. The requirement and conditions have been clarified. This image shows a workspace example.









Firstly I defined these user parameters for the constants.
TILE_WIDTH (Float): horizontal lenghth of a tile
TILE_HEIGHT (Float): vertical length of a tile
TILE_COLUMNS (Integer): number of columns of tiles

The JPEG reader reads every image, the Counter appends 0-based sequential number (_count) to each raster feature, and the RasterPropertiesExtractor extracts raster properties.
"_count" and these two properties will be used in the next step.
_num_columns: width of the image (pixels)
_num_rows: height of the image (pixels)

The AttributeCreator calculates values which should be written into a world file. Here I assigned the values to elements of a list attribute named "text_line_data{}".

The AttributeKeeper removes unnecessary attributes, and the GeometryRemover removes the geometry (raster). These transformers are not essential, but removing unnecessary objects is effective to reduce memory usage especially when exploding the feature.

The ListExploder explodes the feature to create features each of which has a text line.
Finally, the TEXTLINE writer with fanout option writes the text lines into each world file for every image. This image shows the fanout setting.














All the transformers and the TEXTLINE writer can be replaced with a PythonCaller.
-----
# Python Script Example: Create World Files to Arrange Images in Tiles
import fmeobjects

class WorldFileCreator(object):
    def __init__(self):
        # Constants (user parameter values)
        tileWidth = float(FME_MacroValues['TILE_WIDTH'])
        tileHeight = float(FME_MacroValues['TILE_HEIGHT'])
        tileColumns = int(FME_MacroValues['TILE_COLUMNS'])
     
        # Functions for calculating parameter values which will be written into a world file.
        # Every function receives same arguments, and returns a parameter value.
        # w: image width (pixels), h: image height (pixels)
        # List of lambda function objects can be used effectively in this case.
        self.count = 0 # Counter
        self.func = [lambda w, h: tileWidth / w,
                     lambda w, h: 0.0,
                     lambda w, h: 0.0,
                     lambda w, h: -tileHeight / h,
                     lambda w, h: tileWidth * (self.count % tileColumns + 0.5 / w),
                     lambda w, h: -tileHeight * (self.count / tileColumns + 0.5 / h)]
     
    def input(self,feature):
        # Ignore other than raster geometry.
        if feature.getAttribute('fme_type') != 'fme_raster':
            return
         
        # Create output world file path based on the JPEG file path.
        path = feature.getAttribute('fme_dataset')
        if path[-4:].lower() == '.jpg':
            path = '%s.wld' % path[:-4]
        elif path[-5:].lower() == '.jpeg':
            path = '%s.wld' % path[:-5]
        else:
            return
         
        # Call FME Function "@RasterProperties(RASTER)".
        # This function brings the same result as the RasterPropertiesExtractor.
        feature.performFunction('@RasterProperties(RASTER)')
        w = float(feature.getAttribute('_num_columns')) # image width (pixels)
        h = float(feature.getAttribute('_num_rows')) # image height (pixels)
     
        # Write a world file (6 lines), then increment the counter.
        wld = open(path, 'w')
        wld.writelines(['%.8f\n' % f(w, h) for f in self.func])
        wld.close()
        self.count += 1
         
    def close(self):
        pass
-----

2014-03-25

FME Function and PythonCaller

(FME 2014 build 14235)

I tested some FME Functions using the TclCaller in the previous article, but any FME Function can be called also in a Python script embedded in a PythonCaller.
FMEFeature.performFunction() method can be used to call any FME Function on the feature.
Here I give examples of Python edition.

Example 1: Merge Multiple List Attributes
Assume input feature has two list attributes named "_list1{}", "_list2{}". This script creates a new list attribute named "_merged{}" by merging them and removes the original lists.
-----
import fmeobjects
def mergeLists(feature):
    feature.performFunction('@MergeLists(_merged{},_list1{},_list2{})')
    feature.performFunction('@RemoveAttributes(_list1{},_list2{})')
-----

Example 2: Remove Duplicate Vertices from Polygon
This script identifies geometry type of input feature and removes duplicate vertices when the type is fme_polygon.
-----
import fmeobjects
def removeDuplicateVertices(feature):
    if feature.performFunction('@GeometryType()') == 'fme_polygon':
        feature.performFunction('@GeometryType(fme_polygon)')
-----

Example 3: Set Z Values to Coordinates
Assume input feature has a list attribute named "_z{}" storing numeric values. This script set Z values stored in the list to coordinates of the feature when number of the list elements is equal to number of the coordinates.
-----
import fmeobjects
def setZValues(feature):
    n = int(feature.performFunction('@NumCoords()'))
    # The line above can be replaced with:
    # n = feature.numCoords()
    if 0 < n and n == int(feature.performFunction('@NumElements(_z{})')):
        feature.performFunction('@Dimension(2)')
        # The line above can be replaced with:
        # feature.setDimension(fmeobjects.FME_TWO_D)
        feature.performFunction('@ZValue(_z{})')
        feature.setAttribute('_result', 'success')
    else:
        feature.setAttribute('_result', 'failure')
-----

I've never used the FMEFunctionCaller transformer, performFunction method (Python) and FME_Execute procedure (Tcl) in any practical workspace, since the documentation about FME Functions had not been accessable.
But now, the documentation has become available. FME Functions could be used effectively in some cases.

2014-03-22

FME Function and TclCaller

(FME 2014 build 14235)

I discovered by chance that this documentation has been enabled.
> FME Factory and Function Documentation

In a workspace, any FME Function can be called with the FMEFunctionCaller transformer. If the function returns a value, it can be also used in an expression for value setting of many transformers such as AttributeCreator, ExpressionEvaluator etc.. And also, as I mentioned before, they can be called directly from a Tcl script embedded in a TclCaller.
-----
2014-03-25: Of course any FME Function can be also called using the PythonCaller.
See the next article > FME Function and PythonCaller
-----

I tested some functions with TclCaller. Use FME_Execute proc. to call a function in the script.
The syntax is:
FME_Execute <function name without @> [<arg1> <arg2> ... ]

Example 1: Merge Multiple List Attributes
Using @MergeLists function, two or more list attributes can be merged at once.
@RemoveAttributes function removes all the specified attribute(s) including list.
Assume input feature has two list attributes named "_list1{}", "_list2{}". This script creates a new list attribute named "_merged{}" by merging them and removes the original lists.
-----
proc mergeLists {} {
    FME_Execute MergeLists "_merged{}" "_list1{}" "_list2{}"
    FME_Execute RemoveAttributes "_list1{}" "_list2{}"
}
-----
If you don't need to remove the original lists, the FMEFunctionCaller can be used with this parameter setting.
-----
FME Function: @MergeLists(_merged{},_list1{},_list2{})
-----
# Why isn't there "ListMerger" transformer?

Example 2: Remove Duplicate Vertices from Polygon
@GeometryType function is interesting.
If no argument was given, it returns a geometry type identifier of the feature. e.g. fme_point, fme_line, fme_polygon etc..
If fme_polygon was given as its argument, it removes duplicate vertices from the polygon.
Other than above, this function has also several options.
This script identifies geometry type of input feature and removes duplicate vertices when the type is fme_polygon.
-----
proc removeDuplicateVertices {} {
    if {[string compare [FME_Execute GeometryType] fme_polygon] == 0} {
        FME_Execute GeometryType fme_polygon
    }
}
-----
If geometry type of input feature is always fme_polygon, the FMEFunctionCaller can be used with this parameter setting.
-----
FME Function: @GeometryType(fme_polygon)
-----

Example 3: Set Z Values to Coordinates
@NumCoords returns number of coordinates of the feature.
@NumElements <list name> returns number of the list elements.
@Dimension 2|3 forces the feature to 2D|3D.
@ZValue <list name> set Z values stored in the list to coordinates of the feature.
Assume input feature has a list attribute named "_z{}" storing numeric values. This script set Z values stored in the list to coordinates of the feature when number of the list elements is equal to number of the coordinates.
-----
proc setZValues {} {
    set n [FME_Execute NumCoords]
    if {0 < $n && $n == [FME_Execute NumElements "_z{}"]} {
        FME_Execute Dimension 2
        FME_Execute ZValue "_z{}"
        return "success"
    } else {
        return "failure"
    }
}
-----
In my testing, @ZValue function did nothing if coordinates had Z values already. So I've used @Dimension function to force the feature to 2D beforehand.

Interesting?

2014-03-17

Synchronize Multiple User Parameter Values

(FME 2014 build 14235)

The green shape is a 3D polygon. Consider calculating its perimeter and area in 2D or 3D.













The calculation itself is easy. Just use a LengthCalculator and an AreaCalculator.







Parameter Settings
TransformerParameter NameChoice for 2DChoice for 3D
LengthCalculatorLength Dimension23
AreaCaluculatorTypePlane AreaSloped Area

Now, assume that there is a requirement that either 2D or 3D has to be determined through a user parameter at run-time.
It's easy to create two published parameters linked to the two transformer parameters separately, but the user will have to always be careful of their consistency. Ideally, it's better that the two parameter values can be specified simultaneously through setting just one published parameter. In other words, the two parameter values should be synchronized.

If value choices of the two transformer parameters were same, you could link both of them to the same user parameter. However, as shown in the table, those are different in fact.
In such a case, I would create a published parameter and a private scripted parameter so that value of the private parameter would be determined depending on value of the published parameter.
For example:

Published Parameter
Type: Choice
Name: DIM
Configuration: 2%3

Private Parameter
Type: Scripted (Python) or Scripted (Tcl)
Name: AREA_TYPE
-----
# Parameter Value (Python Script) Example
return 'Plane Area' if FME_MacroValues['DIM'] == '2' else 'Sloped Area'
-----
# Parameter Value (Tcl Script) Example
if {$FME_MacroValues(DIM) == 2} {set t "Plane Area"} else {set t "Sloped Area"}
return $t
-----

This Tcl script is also available.
-----
array set a {2 "Plane Area" 3 "Sloped Area"}
return $a($FME_MacroValues(DIM))
-----

And then, specify $(DIM) to "Length Dimension" of the LengthCalculator, $(AREA_TYPE) to "Type" of the AreaCaluclator. User doesn't need to be careful of the consistency of the two parameter values no longer.

Other than the LengthCalculator and AreaCalculator, there are several transformers which have a parameter choosing 2D or 3D. Parameter names and value choices are various.
TransformerParameter NameChoice for 2DChoice for 3D
MeasureGeneratorLength Dimension23
SnipperMeasurement Mode2D3D
GeometryValidator
(Duplicate Consecutive Points)
Check Z ValuesNoYes
etc.

=====
At first, I expected that I could do that with "Choice with Alias" parameter, rather than script.
-----
Type: Choice with Alias
Name: AREA_TYPE
Configuration: 2,Plane<space>Area%3,Sloped<space>Area
Default Value: $(DIM)
-----
But unfortunately value of this parameter became the same value as $(DIM), i.e. 2 or 3.
Why not?

2014-03-15

Transform 3D Line into Pipe-like Solids

(FME 2014 build 14235)

From an actual job which I've completed this week. The requirement was to transform 3D polyline like this image into pipe-like solids.









Through the following manipulations, cylindric solids along the line can be created.

Step 1










Chopper divides the polyline into individual line segments.
LengthCaluculator calculates 2D length of the line segment (_length).
Two CoordinateExtractor extract coordinates of start and end points (x0, y0, z0), (x1, y1, z1).
2DEllipseReplacer and 3DForcer create horizontal circle whose center is located at (x0, y0, z0).









Step 2













AttributeCreator calculates components of vector representing direction of the line segment.
vx = x1 - x0
vy = y1 - y0
vz = z1 - z0
3DRotator rotates the circle so that the face turns to direction of the vector. PI is a user parameter whose value is the circular constant.
Note: There is a bug on the 3DRotator (Custom Axis mode, FME 2014 build 14235), it doesn't work correctly when more than one axis direction has been specified. But it works fine if the rotational origin was (0, 0, 0). So I used two Offsetter transformers in the workflow as a workaround for the bug. After the bug having been fixed, remove the Offsetters and specify (x0, y0, z0) to "Origin X, Y, Z" of the 3DRotator.
Regarding the bug, see also here (Community members only) > Chatter: 3DRotatorProblem









Extruder extrudes the circle towards direction of the vector to create cylindric solid. Done!









Ideally I wanted to resolve the gaps which occur at the joints between cylinders, but the solution has not been found out yet. Similar issue had been discussed in the Community last year.
I'm looking forward to Safe's solution ;)

2014-03-08

Classify Polygon Shape with Python

(FME 2014 build 14235)

From this thread > Community Answers: Parcels geometric shape

A Python script example for the task.
I think the getVertices function in this script can be used for general purpose.
-----
# PythonCaller Script Example: Classify Polygon Shape
# 2013-03-09 Added triangle classification.
# Classify a polygon into triangle, quadrilateral, and other.
# Triangle will be further classified into right, isosceles, and equilateral.
# Quadrilateral will be further classified into trapezoid, parallelogram, rectangle, and square.
import fmeobjects

def classifyPolygon(feature):
    shape = 'Invalid'
    g = feature.getGeometry()
    if isinstance(g, fmeobjects.FMEPolygon) and g.isBoundaryLinear():
        vertices = getVertices(feature)
        if len(vertices) == 3:
            shape = ','.join(classifyTriangle(vertices))
        elif len(vertices) == 4:
            shape = ','.join(classifyQuadrilateral(vertices))
        elif 4 < len(vertices):
            shape = 'Other'
    feature.setAttribute('_shape', shape)

X, Y, PRECISION = 0, 1, 1.0e-6
sqrLength = lambda a: a[X]**2 + a[Y]**2
isNearZero = lambda v: abs(v) < PRECISION
isZeroLength = lambda a: isNearZero(sqrLength(a))
isSameLength = lambda a, b: isNearZero(sqrLength(a) - sqrLength(b))
isParallel = lambda a, b: isNearZero(a[X] * b[Y] - a[Y] * b[X]) # outer product
isPerpendicular = lambda a, b: isNearZero(a[X] * b[X] + a[Y] * b[Y]) # inner product

# Remove excess coordinates and return tuple (x, y) list of polygon vertices.
def getVertices(feature):
    coords = feature.getAllCoordinates()
    if feature.getDimension() == fmeobjects.FME_THREE_D:
        coords = [(x, y) for x, y, z in coords]
    if len(coords) < 3:
        return coords
    x0, y0 = coords[0][X], coords[0][Y]
    x1, y1 = coords[1][X], coords[1][Y]
    vertices = [(x0, y0)]
    for x2, y2 in coords[2:]:
        a = (x1 - x0, y1 - y0)
        b = (x2 - x1, y2 - y1)
        if not isZeroLength(a) and not isZeroLength(b) and not isParallel(a, b):
            vertices.append((x1, y1))
            x0, y0 = x1, y1
        x1, y1 = x2, y2
    return vertices

# Return string list of triangle class names.
def classifyTriangle(vertices):
    # Create verctors of three edges.
    p1, p2, p3 = vertices[0], vertices[1], vertices[2]
    v1 = (p2[X] - p1[X], p2[Y] - p1[Y])
    v2 = (p3[X] - p2[X], p3[Y] - p2[Y])
    v3 = (p1[X] - p3[X], p1[Y] - p3[Y])
 
    # Classify the triangle.
    shape = ['Triangle']
    if isPerpendicular(v1, v2) or isPerpendicular(v2, v3):
        shape.append('Right')
    if isSameLength(v1, v2):
        shape.append('Isosceles')
        if isSameLength(v2, v3):
            shape.append('Equilateral')
    elif isSameLength(v2, v3):
        shape.append('Isosceles')
    return shape

# Return string list of quadrilateral class names.
def classifyQuadrilateral(vertices):
    # Create vectors of four edges.
    p1, p2, p3, p4 = vertices[0], vertices[1], vertices[2], vertices[3]
    v1 = (p2[X] - p1[X], p2[Y] - p1[Y])
    v2 = (p3[X] - p2[X], p3[Y] - p2[Y])
    v3 = (p4[X] - p3[X], p4[Y] - p3[Y])
    v4 = (p1[X] - p4[X], p1[Y] - p4[Y])
 
    # Classify the quadrilateral.
    shape = ['Quadrilateral']
    if isParallel(v1, v3):
        shape.append('Trapezoid')
        if isParallel(v2, v4):
            shape.append('Parallelogram')
            if isPerpendicular(v1, v2):
                shape.append('Rectangle')
                if isSameLength(v1, v2):
                    shape.append('Square')
    elif isParallel(v2, v4):
        shape.append('Trapezoid')
    return shape
-----

Calculate Spacing of Grid Points

(FME 2014 build 14235)

I got a dataset which contains 3D grid points.













The mission was to create a raster based on the data. As a requirement, each cell of the resultant raster would have to contain the original grid point at its center. That is, I had to create a raster whose cell spacing matches with the spacing of the grid points.
It was known that x, y spacings are constant, but the exact values were unknown. So I needed to calculate the spacings based on the point geometries.

At first, inspecting the data with the Data Inspector, I determined approximate ranges, and defined them as user parameters.
APPROX_MIN_X_SPACING, APPROX_MAX_X_SPACING
APPROX_MIN_Y_SPACING, APPROX_MAX_Y_SPACING

Then, I created a workflow like this to calculate the spacings.










In the actual dataset, x-spacing and y-spacing were the same value, but the workflow works even if they are different values.
There might be a quick function which calculates the spacing of grid points in FME, but I wasn't able to find it. If you know, tell me that!

2014-03-01

Set Measure Values to Line Vertices

Think about setting measure values to every vertex of line geometries. Assume that the measure value has to be distance from start node of the line which the vertex belongs to.

If the coordinate system for measuring is same as the source coordinate system, a MeasureGenerator can be used simply.
If not, a quick way is to perform reprojection before and after the MeaureGenerator.

Reprojector -> MeasureGenerator -> Reprojector

But output line geometry may not be strictly same as input line, since interpolation is performed in reprojection process in general. Although the error caused by reprojection is slight amount, cannot be avoided.
If such an error will not be allowed, this workflow would be a workaround.









The pattern of the workflow - adding temporary ID, branching feature flow into multiple streams, and merging them after processing - is used in many cases. I think there are many general patterns which frequently appear in FME workspaces. I personally call them "Standard methods".
The pattern above is typical one. I call it "Counting-Branching-Merging" method, provisionally.
How do you call it?

=====
2014-03-04: I got a great comment on this subject. That is a suggestion regarding another workaround with the GeometryExtrator and GeometryReplacer like this.















In my quick test for a large dataset, this method was apparently more efficient in both memory usage and processing speed (Geometry Encoding: FME Binary). And also the original geometries have been restored exactly. I'm going to research further more in another opportunity.
I've never noticed such an effective usage of GeometryExtractor / Replacer. I would add the pattern of "ExtractGeometry-Processing-RestoreGeometry" to my "Standard methods".
Thanks for the input!

P.S. It is a basic usage of the GeometryExtractor / Replacer. I was blind ...
"This transformer is often used to make a copy of the feature's geometry into an attribute before some temporary geometry change is made, so that it can later be restored."
-- Help on the GeometryExtractor, FME 2014
"This transformer is typically used to restore geometry previously extracted into an attribute by the GeometryExtractor."
-- Help on the GeometryReplacer, FME 2014

-----
If "ExtractGeometry-Processing-RestoreGeometry" method would be used generally in many scenarios, it might be more convenient that the GeomeryExtractor and GeometryRepalcer have an optional parameter "Coordinate System Attribute", so that the method can extract and restore coordinate system without using the CoordinateSystemExtractor / Setter if necessary.