Version 25 (modified by Vishrut Mehta, 8 years ago) ( diff )


Project: Full-Text Search

Name : Vishrut Mehta
Mentor: Pat Tressel


The proposal for the project is here:


This project draws ideas from the Blueprints below:

Meetings And Discussions

Weekly Meeting : Tuesday and Saturday 04:30 UTC

Venue : IRC
Nick - vishrut009

Google Group Discussions :

Description of Work Done

  • Sunburnt
    • The script Attached below installs the dependencies and also configures and installs Apache Solr and Sunburnt
  • Enabling text search:

-> Uncomment the following line in models/

# Uncomment this and set the solr url to connect to solr server for Full-Text Search
settings.base.solr_url = ""
  • Asynchronously Indexing and Deleting Documents:
    • The code for asynchronously indexing documents is in models/
    • Insertion: The code will first insert the document into the database. Then in callback onaccept it will index those documents calling the document_create_index() function from models/ . The following code should be added for enabling Full Text search for documents in any modules. The example is there, you can see modules/s3db/ in document_onaccept() hook.
    • Deletion: The code will first delete the record from the database table, then will select that file and will delete it from Solr also, by deleting its index which is stored in solr server. You can look for the code in modules/s3db/ in document_ondelete() hook.
    • In model()
              if settings.get_base_solr_url():
                  onaccept = self.document_onaccept # where document_onaccept is the onaccept hook for indexing
                  ondelete = self.document_ondelete # where document_onaccept is the onaccept hook for deleting
                  onaccept = None
                  ondelete = None
    • In onaccept()
          def document_onaccept(form):
              vars = form.vars
              doc = vars.file # where file is the name of the upload field
              table = current.db.doc_document # doc_document is the tablename
                  name = table.file.retrieve(doc)[0]
              except TypeError:
                  name = doc
              document = json.dumps(dict(filename=doc,
                                        tablename="doc_document", # where "doc_document" is the name of the database table
                                   args = [document])
    • in ondelete():
          def document_ondelete(row):
              db = current.db
              table = db.doc_document # doc_document is the tablename
              record = db( ==, # where file is the name of the upload field
                                                     limitby=(0, 1)).first()
              document = json.dumps(dict(filename=record.file, # where file is the name of the upload field
                                   args = [document])

Full Text Functionality

  • The full text search functionality is integrated in modules/s3/, the fulltext() does the work.
  • The flow is: First the text query goes to transform() function, which would split the query recursively and then transform the query.
  • After transforming, the query with text operator will go to fulltext() function and would search for the keywords in the indexed documents.
  • It will retrieve document ids and then covert into a belongs query.
SMART Goal Measure Status
Explore Pylucene Installed and configured on demo server Completed
Scripts for indexing and search in pylucene Scripts working on the demo server Completed
Explore Apache Solr and Sunburnt Installed both on demo and local server Completed
Scripts for indexing and search for sunburnt Working scripts for sunburnt ready Completed
Asynchronously Indexing and Deleting Document Implemented & Integrated in Sahana Eden Completed
Install Script foe Installing and Configuring Solr and sunburnt Below is the link of the script Completed
Designing the Full-Text search functionality implementation Discussed with Dominic and Fran Completed
Implementation of fulltext() function in Successfully implemented with Error handling Completed
Implemented a transform() function to transform a text query to belong query Successfully implemented with Error handling Completed
Unit tests for all cases(solr un/available, query(), call() ) Implemented the unit tests for s3resource Almost Done
Generic Indexing onaccept() hook, can be integrated in any module Implemented and tested on doc_document Completed

Attachments (1)

Download all attachments as: .zip

Note: See TracWiki for help on using the wiki.