Creating, Updating, Deleting documents in a Lucene Index with symfony

Previously we covered an all-at-once approach to indexing objects in your symfony app. But for some reason, people find the need to allow users to sign up, or change their email addresses and then all of a sudden our wonderful Lucene index is out of date.

Here lies the strength of using Zend Search Lucene in your app, you can now get the flexibility of interacting with a Lucene index, no matter how it was created and add, update and delete documents to it.

The last thing you want to do is have a cron job in charge of making sure your index is always up to date by reindexing regularly. This is an inelegant and inefficient process.

A smarter method would be to trigger an update of the index each time you update your database. Luckily the ORM layer allows us to do this using objects (in our case Propel objects).

If we look at our user example from before, we did set ourselves up to easily do this using our User::generateZSLDocument() function, which did most of the heavy lifting.

We can make a few small changes to the User class:

We have an attribute called $reindex. When it is false we don’t need to worry about the index. When something significant changes, like an update to your name or email address, then we set $reindex to true. Then when we save with an overridden save method:

Now we’ve got the exact same data that we created during our original indexing. This handled creating and updating object, but we miss updating the index when deleting objects.

Luckily we already made a function User::removeFromIndex() to remove any related documents from the index, so our delete function can be pretty simple:

Read full post
The Lucene Search Index and symfony

[tags]Zend, Zend Search Lucene, Search, Lucene, php, symfony, zsl, index[/tags]

Read full post
MinneBar

[tags]Minnebar, symfony, barcamp, frameworks, php, php5[/tags]

Read full post
Tips for symfony and Subversion

[tags]symfony, subversion[/tags]

There’s some tricks you can do to running a symfony project with subversion:

Ignoring files in cache/ and log/

The first thing you can do (and this is well documented in the askeet tutorial) is ignore files in cache/ and log/. These files are specific to each instance of your app and don’t contain anything that needs to be in version control.

Run the following:

cd $SF_ROOT
rm -rf log/* cache/*
svn propedit svn:ignore log
svn propedit svn:ignore cache

svn propedit will bring up a text editor, in both instances you want to save the following:

*

Ignore other auto-generated files

Eric Sink wrote an excellent tutorial on source control. In his chapter on repositories he recommends checking in only hand edited source code. If a property file generates another file, check in the property file, not the auto-generated result. This not only keeps your repository clean, it prevents a lot of unnecessary check-ins.

If you use propel for your ORM layer there are a few files you can ignore using svn propedit svn:ignore {dirname}.

In $SF_ROOT/config we can ignore:

*schema-transformed.xml

These are xml files that propel generates from schema.xml (or schema.yml).

In $SF_ROOT/data/sql we can ignore:

lib.model.schema.sql
plugins.*.sql
sqldb.map

These are created from schema.xml (or schema.yml) as well.

The real savings will come with your model. The propel model creates customizable php classes in lib/model which inherit from auto-generated files in lib/om there are also auto-generated map files in `lib/map’

We can run from $SF_ROOT:

svn propedit svn:ignore lib/model/om
svn propedit svn:ignore lib/model/map

and enter

*

for both properties.

If you’ve mistakenly checked in some of these files you will need to remove them from your repository via svn delete.

Linking the symfony Library

I prefer to embed the symfony library into each of my symfony apps rather than relying on a shared PEAR library. This lets me run multiple versions of symfony without much fuss. With subversion we can use the svn:externals property to directly link our app with the symfony subversion repository.

At first this sounds like danger, but externals can be linked to specific revisions. However, the symfony team tags their repository with version numbers. To get this to work we need to do 3 things. (UPDATE: See Fabien’s comment about using the lib/vendor directory)

  1. Modify config/config.php to look for symfony internally. Just open it up and change it so it says this:
  2. Run svn propedit svn:externals lib from $SF_ROOT and enter:

     symfony http://svn.symfony-project.com/tags/RELEASE_1_0_2/lib/ or whatever version of symfony you want to link to, at the time of this post, `RELEASE_1_0_2` is fairly fresh.
    
  3. Run svn propedit svn:externals data from $SF_ROOT and enter:

     symfony http://svn.symfony-project.com/tags/RELEASE_1_0_2/data/ or whatever version of symfony you want to link to, at the time of this post.
    

Now when you do svn update you’ll have the symfony library all linked up. Furthermore this keeps all the developers on the same version of symfony.

Also you may want to start running symfony using ./symfony versus symfony. The former looks at your configuration settings to determine which symfony command to use, the latter is generally linked to your system wide command (which is generally the PEAR installed command).

Linking to symfony Plugins

I have my hands in a number of symfony plugins because I work on a lot of projects which tend to share a lot of similar functionality. Many of the plugins are in early stages of development, so I find it helpful to have them linked from svn as well. This way I can get the benefits of any new functionality and if the occasion should arise, I can contribute any useful changes I make.

To link to the plugins you run svn propedit svn:externals plugins and enter one plugin per line in the following format:

{plugin_name} -r{REVISION} {URL}

For one of my projects I use:

sfPrototypeTooltipPlugin http://svn.symfony-project.com/plugins/sfPrototypeTooltipPlugin
sfGuardPlugin http://svn.symfony-project.com/plugins/sfGuardPlugin
sfZendPlugin http://svn.symfony-project.com/plugins/sfZendPlugin

I’ve omitted the revision, because I live dangerously and want to use the latest $HEAD.

Read full post
Hitask

layout: post title: HiTask site: spindrop tags: [spindrop, web2.0, hiTask, tasks, gtd] — [hiTask]: http://hitask.com/

Read full post
sfZendPlugin

[tags]Zend, Zend Search Lucene, Search, Lucene, php, symfony, zsl, plugins[/tags]

Read full post
Minneapolis Cable Internet for less than Comcast charges ;)

You can signup for Earthlink which uses comcast of $30/mo for 6 months $45/mo after… and you get 7000 miles from NWA.

Read full post
Creative Friends

[tags]photography, design, pretty[/tags]

Read full post
Caching REST with sfFunctionCache

[tags]geocoding, caching, REST, symfony, Cache_Lite, php, cache, sfFunctionCache[/tags]

For reviewsby.us we do a lot of geocoding. To facilitate we use Yahoo! Geocoding API. This helps us normalize data, obtain latitude and longitude, interpret location specific searches.

These REST queries happen a lot and will continue to happen, but this data that Yahoo! provides is fairly static. We’re basically querying a database of sorts. So it makes sense that we should cache this data.

We’ll demonstrate how to cache these queries using symfony’s sfFunctionCache class.

I wrote a wrapper (I’ll release it as a plugin if requested) for the Geocoding API, the bulk of the work (the REST call) occurs in a function called doQueryGIS:

The call to this function is always wrapped with queryGIS:

This wrapper creates a sfFunctionCache objet and calls the function and caches it for subsequent queries.

What this means is once Yahoo! teaches reviewsby.us that India is located at (25.42°, 77.830002°) and that the precision is ‘country’ we remember it in the future.

These features will be incorporated into future versions of reviewsby.us.

Read full post