Tuesday, March 21, 2017

Add mobile support to your Moodle plugin - part two

Part Two - Exploring the Moodle Mobile Add-on

In the first part of this series, I set up a development environment on my local machine. I am going to use this same environment to begin coding the mobile add-on portion of my Moodle questionnaire plug-in.

The documentation says I need to develop the mobile side as a standard Moodle Mobile add-on and then package it as a remote add-on, with four steps:
  1. Develop the required Moodle Web Services
  2. Develop a standard Moodle Mobile add-on
  3. Package the Moodle Mobile add-on as a remote add-on
  4. Include the remote add-on in your Moodle plugin
The first two steps are where my work will start. The second two steps are what I will need when I am ready to distribute what I have developed for anyone to use in their mobile apps with their Moodle sites. I am also going to start with step two, as this will help me understand what is needed in the mobile portion and what is needed from the Moodle plugin portion.

Recall, that when I try to access my plugin in the mobile app now, I get the screen:


My goal, for the start, is to provide enough code so that screen changes.

To start, I look at the current structure of the mobile app on my local server. Within it is a www/addons directory that contains all of the core mobile addons. I can see, that like a Moodle site, there is a mod subdirectory, and within that, a number of subdirectories corresponding to each of the supported core activity plugins.

For ease of development, I am going to add my plugin to the same location. That way, I can update my code within the mobile app, and access it with the browser, to see my work. Essentially, my local files will stand-in as the mobile app, and the questionnaire mobile addon will act as a core activity plugin.

For purposes of code storage, I am going to add an addons subdirectory to my questionnaire repository and add the code that will go into the mobile addon there. This code will not be required, nor run, on a Moodle site, but I want to keep it together with my Moodle plugin for development ease.

Through a productive discussion with the mobile development team, I have determined that getting my add-on registered with the mobile app requires code that registers a content handler. In the add-on structure, this happens in the main.js file in a config statement (full code here):
.config(function($mmCourseDelegateProvider) {
    $mmCourseDelegateProvider.registerContentHandler('mmaModQuestionnaire', 'questionnaire', '$mmaModQuestionnaireHandlers.courseContent');
});
Essentially, the mobile framework translates this code such that it knows there must be a services subdirectory and within that a file named handlers.js that contains a courseContent function. I create that structure with a very simple courseContent function as follows (full code here):
self.courseContent = function() {
    var self = {};
    /**
     * Whether or not the module is enabled for the site.
     *
     * @return {Boolean}
     */
    self.isEnabled = function() {
        return true;
    };
    return self;
};
In conversations with the developers, I have determined that I need an isEnabled function, that returns true if the plugin is enabled, for my addon to be acknowledged and not display the "plugin that is not yet supported" message. For now, I just send true with no logic confirming that. Eventually, this will call a web service from the actual questionnaire plugin on the Moodle site to determine this. I also know that when I run my local version of the mobile app using the ionic server, it will build my addons code into the app's main code file, www/build/mm.bundle.js.

After dropping these two files into my mobile app's www/addons/mod/questionnaire directory, I run the ionic server (removed some messages for brevity):
ionic serve --browser chromium
Running 'serve:before' gulp task before serve
[13:43:32] Starting 'build'...
[13:43:32] Starting 'sass-build'...
[13:43:32] Starting 'lang'...
[13:43:32] Starting 'watch'...
[13:43:41] Finished 'watch' after 9.04 s
[13:43:47] Finished 'sass-build' after 15 s
[13:43:47] Starting 'sass'...
[13:43:48] Finished 'lang' after 15 s
[13:43:49] Finished 'sass' after 1.92 s
[13:43:50] Finished 'build' after 18 s
[13:43:50] Starting 'config'...
[13:43:50] Finished 'config' after 17 ms
[13:43:50] Starting 'default'...
[13:43:50] Finished 'default' after 6.02 μs
[13:43:50] Starting 'serve:before'...
[13:43:50] Finished 'serve:before' after 3.15 μs
Running live reload server: http://localhost:35729
Watching: www/**/*.html, www/build/**/*, www/index.html, !www/lib/**/*
√ Running dev server:  http://localhost:8100
Ionic server commands, enter:
  restart or r to restart the client app from the root
  goto or g and a url to have the app navigate to the given url
  consolelogs or c to enable/disable console log output
  serverlogs or s to enable/disable server log output
  quit or q to shutdown the server and exit
ionic $
I can see a lot of build activity happening. I then point my browser to the server instance and see a lot more activity kick in. When I access my course, and then click on my questionnaire activity, I no longer get the "plugin that is not yet supported" message:


I don't get the questionnaire displayed, but I didn't expect to. I have at least learned how to impact the mobile app with my addon.

I also take a look at the www/build/mm.bundle.js to see if it now contains the code I created for the questionnaire addon. A quick search of the code reveals that it does. So I have confirmed that the mobile app has successfully picked up the questionnaire addon code and added it to the app.

In the next post, I will flesh out more of the mobile add-on code, and begin to look what the web services needed on the questionnaire plugin side.

Friday, March 3, 2017

Add mobile support to your Moodle plugin - part one

Part One - Setting up a Development Environment

With version 3 of Moodle came the Moodle Mobile app. Out of the box, most of the core Moodle plugins came supported. But, third-party plugins do not function by default in the mobile app.

In Moodle 3.1, the mobile app added support for "Remote add-ons". This support allows plugin providers to add support so that their plugin can function within the mobile app.

Now, I should point out that Moodle is currently working on a much simpler system for adding mobile support to your plugin, but it will be a while before it is ready. When it is ready, the work I am about to undertake will likely be moot.

This post will be the first in a series where I will attempt to add mobile support for my questionnaire module. I say "attempt", because this will be using some technologies that I am not familiar with so I will be learning as I go. I won't guarantee success, but I will document the efforts.

To start with, the main developer documents for Moodle Mobile are here. Looking through that list, I'm going to start with setting up a development environment on my Mac.

I already have Chrome installed on my Mac, so I can skip that step. The next step it says is to install "Node.js". Keep in mind, I don't really know what I am doing here (yet), so I'm not sure how this comes into play, but I'm going to follow instructions. :-)

The documentation suggest using Macports to install Nodejs. It also says I need to use version "v0.12.7". But, it looks like Macports only has versions from 4.* to 7.*. So, Macports is out. The documentation also provides a direct download link for "v0.12.7", and it has a Mac "pkg". So I download that and run the Mac installer. As a result, I have "node" installed at /usr/local/bin/node and "npm" installed at /usr/local/bin/npm.

The next step asks me to run the command npm cache clean. When I do this, I get a series of errors. which seem to indicate permission problems. Likely I need to elevate the permission level this command runs at, so I add a sudo to the front of the command and try again. The new command, sudo npm cache clean works fine. I'm going to assume I will need sudo for the rest of the commands in the document as well.

Next, I am asked to install "ionic". I excute the command:
sudo npm install -g cordova ionic
I get some issues displayed:

npm WARN engine cordova@6.5.0: wanted: {"node":">=4.0.0"} (current: {"node":"0.12.7","npm":"2.11.3"})
npm WARN engine request@2.79.0: wanted: {"node":">= 4"} (current: {"node":"0.12.7","npm":"2.11.3"})
npm WARN deprecated node-uuid@1.4.7: use uuid module instead
/usr/local/bin/cordova -> /usr/local/lib/node_modules/cordova/bin/cordova
/usr/local/bin/ionic -> /usr/local/lib/node_modules/ionic/bin/ionic
cordova@6.5.0 /usr/local/lib/node_modules/cordova
├── underscore@1.7.0
├── q@1.0.1
├── nopt@3.0.1 (abbrev@1.1.0)
├── update-notifier@0.5.0 (is-npm@1.0.0, semver-diff@2.1.0, chalk@1.1.3, string-length@1.0.1, repeating@1.1.3, configstore@1.4.0, latest-version@1.0.1)
├── insight@0.8.4 (object-assign@4.1.1, uuid@3.0.1, lodash.debounce@3.1.1, async@1.5.2, chalk@1.1.3, configstore@1.4.0, os-name@1.0.3, tough-cookie@2.3.2, request@2.79.0, inquirer@0.10.1)
├── cordova-common@2.0.0 (cordova-registry-mapper@1.1.15, ansi@0.3.1, semver@5.3.0, osenv@0.1.4, underscore@1.8.3, q@1.4.1, unorm@1.4.1, shelljs@0.5.3, minimatch@3.0.3, glob@5.0.15, bplist-parser@0.1.1, elementtree@0.1.7, plist@1.2.0)
└── cordova-lib@6.5.0 (valid-identifier@0.0.1, opener@1.4.1, cordova-registry-mapper@1.1.15, properties-parser@0.2.3, nopt@3.0.6, unorm@1.3.3, shelljs@0.3.0, semver@4.3.6, glob@5.0.15, dep-graph@1.1.0, xcode@0.9.1, elementtree@0.1.6, init-package-json@1.9.4, cordova-serve@1.0.1, request@2.47.0, tar@1.0.2, cordova-fetch@1.0.2, aliasify@1.9.0, plist@1.2.0, cordova-js@4.2.1, cordova-create@1.0.2, npm@2.15.11)

ionic@2.2.1 /usr/local/lib/node_modules/ionic
└── @ionic/app-generators@0.0.3
But they seem to be warnings about outdated "node". I know we are using an older version of "node", so I believe this will not be an issue.

Now I need to install the "bower" and "gulp" node packages.

Bower:
sudo npm install -g bower
/usr/local/bin/bower -> /usr/local/lib/node_modules/bower/bin/bower
bower@1.8.0 /usr/local/lib/node_modules/bower
Gulp:

sudo npm install -g gulp
npm WARN deprecated minimatch@2.0.10: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
npm WARN deprecated minimatch@0.2.14: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
npm WARN deprecated graceful-fs@1.2.3: graceful-fs v3.0.0 and before will fail on node releases >= v7.0. Please update to graceful-fs@^4.0.0 as soon as possible. Use 'npm ls graceful-fs' to find it in the tree.
/usr/local/bin/gulp -> /usr/local/lib/node_modules/gulp/bin/gulp.js
gulp@3.9.1 /usr/local/lib/node_modules/gulp
├── interpret@1.0.1
├── pretty-hrtime@1.0.3
├── deprecated@0.0.1
├── archy@1.0.0
├── tildify@1.2.0 (os-homedir@1.0.2)
├── minimist@1.2.0
├── v8flags@2.0.11 (user-home@1.1.1)
├── chalk@1.1.3 (escape-string-regexp@1.0.5, supports-color@2.0.0, ansi-styles@2.2.1, has-ansi@2.0.0, strip-ansi@3.0.1)
├── semver@4.3.6
├── orchestrator@0.3.8 (sequencify@0.0.7, stream-consume@0.1.0, end-of-stream@0.1.5)
├── gulp-util@3.0.8 (object-assign@3.0.0, array-differ@1.0.0, lodash._reescape@3.0.0, lodash._reinterpolate@3.0.0, lodash._reevaluate@3.0.0, beeper@1.1.1, array-uniq@1.0.3, replace-ext@0.0.1, dateformat@2.0.0, has-gulplog@0.1.0, fancy-log@1.3.0, vinyl@0.5.3, gulplog@1.0.0, lodash.template@3.6.2, through2@2.0.3, multipipe@0.1.2)
├── liftoff@2.3.0 (lodash.isstring@4.0.1, lodash.isplainobject@4.0.6, rechoir@0.6.2, extend@3.0.0, flagged-respawn@0.3.2, lodash.mapvalues@4.6.0, resolve@1.3.2, fined@1.0.2, findup-sync@0.4.3)
└── vinyl-fs@0.3.14 (strip-bom@1.0.0, graceful-fs@3.0.11, vinyl@0.4.6, defaults@1.0.3, mkdirp@0.5.1, through2@0.6.5, glob-stream@3.1.18, glob-watcher@0.0.6)
Again, there are some warnings about outdated releases. But I'm just going to leave them as is for now.

The next step talks about "Push notifications for Mac", and refers me to "https://cocoapods.org/" and the Moodle tracker item MOBILE-1970. It looks like this will resolve a problem I may have later on when running iOS versions of the app. So, I execute the command:
sudo gem install cocoapods
At the end of that, I have 27 new gems installed in /Library/Ruby/Gems/2.0.0/gems/cocoapods-1.2.0/.

I already have my git clone of the moodlemobile2 repository, so I set my current directory to it. From there, I execute:
sudo npm install
There are a number of warnings and errors that are output as this executes. Most of them appear to be problems with outdated modules. I am going to disregard those for now.

Now I run the following three commands, one after the other:
sudo ionic platform add android@5.1.1
sudo ionic platform add ios@4.1.0
sudo ionic state restore
All three seem to install okay. The only warnings I receive are for outdated dependencies.

So, I execute:
sudo bower install
And I get:
bower ESUDO         Cannot be run with sudo
So, guess I won't be using sudo here.

I rerun the command without sudo, and get a long list of files being installed. All of them went into the local repo below www/lib/.

So, gulp:

gulp
[16:30:43] Using gulpfile ~/www/moodlemobile2/gulpfile.js
[16:30:43] Starting 'build'...
[16:30:43] Starting 'sass-build'...
[16:30:43] Starting 'lang'...
[16:30:48] Finished 'sass-build' after 5.38 s
[16:30:48] Starting 'sass'...
[16:30:49] Finished 'lang' after 5.59 s
[16:30:50] Finished 'sass' after 1.7 s
[16:30:50] Finished 'build' after 7.46 s
[16:30:50] Starting 'config'...
[16:30:51] Finished 'config' after 16 ms
[16:30:51] Starting 'default'...
[16:30:51] Finished 'default' after 5.49 μs
Which again added files below my local repo.

At this point, I think I have everything ready to be able to access the Moodle Mobile app from my local development environment. The links tell me to start Chrome using:

    open -a "Google Chrome" --args --allow-file-access-from-files --disable-web-security --user-data-dir


That seems to just move my Chrome window to the front. Probably because I already had it open? I'll assume it's all good.

Now, to run the ionic server:
ionic serve --browser chromium
******************************************************
 Dependency warning - for the CLI to run correctly,    
 it is highly recommended to install/upgrade the following:  
 Please install your Cordova CLI to version  >=4.2.0 `npm install -g cordova`
 Install ios-sim to deploy iOS applications.`npm install -g ios-sim` (may require sudo)
 Install ios-deploy to deploy iOS applications to devices.  `npm install -g ios-deploy` (may require sudo)
******************************************************
WARN: ionic.project has been renamed to ionic.config.json, please rename it.
WARN: ionic.project has been renamed to ionic.config.json, please rename it.
Multiple addresses available.
Please select which address to use by entering its number from the list below:
 1) 10.120.211.137 (en4)
 2) 10.120.211.40 (en0)
 3) localhost
Address Selection: 3
Selected address: localhost
Running live reload server: http://localhost:35729
Watching: www/**/*, !www/lib/**/*
√ Running dev server:  http://localhost:8100
Ionic server commands, enter:
  restart or r to restart the client app from the root
  goto or g and a url to have the app navigate to the given url
  consolelogs or c to enable/disable console log output
  serverlogs or s to enable/disable server log output
  quit or q to shutdown the server and exit
Since I chose the 'localhost' option I enter 'http://localhost:8100' in my chrome browser window. The browser displays the Moodle Mobile app! I configure it to my local running copy of Moodle, and I'm in business!

Using the simulated mobile browser, I navigate to my test course:


I can see a "questionnaire" instance, so I select that:


And, I see that I really do need to add mobile support to the plugin!

Next posts will be me trying to make this happen.

Friday, February 3, 2017

Adding search to your Moodle plugin - post mortem

Post Mortem - Production ready search

I ended up in a lot of discussions with other Moodle developers on the best way to implement the question content search. Originally, I created a separate search area, but was hampered by the fact that I couldn't reliably keep a record per question linked to a questionnaire instance.

The search API didn't allow me to create my own index other than an integer. If it would have allowed me to construct my own, I could have stored information that identified the questionnaire and question. This resulted in the creation of the Moodle tracker item MDL-57857.

However, as we discussed this particular problem more, it really seemed that if I was searching for questionnaire instances that contained the searched for question content, then the index should really be part of the activity search. This resulted in the update I posted at the end of the last post. While that worked, the return results displayed weren't very attractive and, as was pointed out, would also include question content that may never have been displayed to the use. Questionnaires can choose questions to display based on answers to other questions.

At that point, I realized that the most likely use case for searching for question content would be for users who can access all of the question content of a questionnaire. This would be users who could actually edit the questions and users who could see all of the responses to any question.

So, it made more sense to again separate out the question search area into its own search area. This allows for two important things:

  1. Allows the administrator of the site to determine if they want to even allow question content searching.
  2. Allows the access for this content to be limited to users who have the ability to see all of the question content for a questionnaire.
You can see the final code I decided on here. The question area still stores one record per questionnaire activity, like the activity area, but it uses a different set of access rules. And, since it is a separate area, it can be enabled/disabled globally, and filtered on separately.

I believe that until there is a way to store information in the search index that allows me to go directly to the question within a specific questionnaire, this is the best solution.

Wednesday, February 1, 2017

Adding search to your plugin - Part three

Part Three - Indexing More of your Plugin

In the last post, I added more searchable content to my plugin's search indexing function, so that additional information fields specific to each instance of a plugin were included. Now I'm going to try and add question content to the search indexing.

The documentation tells me that if I'm going to index other activity data, I should use the base_mod class instead of the base_activity class. The base_mod class is located in the core file /search/classes/base_mod.php. Looking at that class, I see that it only contains one method: get_cm. All the rest come from the class it extends, base. The functions I used before, get_recordset_by_timestamp() and get_document() are abstract functions of base. This means I must provide their implementations. I also note that three other abstract functions, check_access(), get_doc_url() and get_context_url() that I will likewise need to provide implementations for. In my previous work, these three functions were provided to me by the base_activity class.

Back to the documentation, the first step it points out is that if I am going to add a new search area for my plugin, I need to provide a name for it in my language strings. Since I am going to index the questions, I add the string:

    $string['search:question'] = 'Questionnaire - questions';

to my lang/en/questionnaire.php file.

Next, I need to provide the indexed content from my questions to the search engine, using the get_recordset_by_timestamp() function. Recall that this function is responsible to construct the necessary database query for all of the content I want available for searching, execute that query, and then return the resulting recordset. For this, I create a new file called 'classes/search/question.php' and create a new question class extending the \core_search\base_mod class.

For questionnaire questions, I will need to join two tables. Question data is contained in a table that contains a reference to a survey id. That survey id is also referenced in the main questionnaire table. Since question data is not accessed outside of a survey context, I will need to create SQL that returns each question with an associated questionnaire and survey.

The function I create looks like this (full file here):

    public function get_recordset_by_timestamp($modifiedfrom = 0) {
        global $DB;

        $sql = 'SELECT qq.id, q.id AS questionnaireid, q.timemodified, ' .
            'q.course AS courseid, q.introformat, qq.name, qq.content ' .
            'FROM {questionnaire} q ' .
            'INNER JOIN {questionnaire_question} qq ON q.sid = qq.survey_id AND ' .
            'qq.deleted = \'n\' ' .
            'WHERE q.timemodified >= ? ' .
            'ORDER BY q.timemodified ASC';

        return $DB->get_recordset_sql($sql, [$modifiedfrom]);
    }

Note that I use the question id field, 'qq.id', as the first field. Moodle requires the first field of a query return to be a unique field that is used as an index for the resulting data array. Since there will be multiple questions per questionnaire and survey record, I can't use those. I will need the questionnaire id field though, so I rename it to 'questionnaireid' in the returned data. The question name and content fields, 'qq.name' and 'qq.content', contain the data that will actually be searched. This function will provide the data that will be indexed by the search function.

Now I need to provide the get_document() function. This is the function that looks at an indexed record returned from a search query and constructs the document object that is displayed on the search results screen. A document object contains a 'title' and 'content' field. I set the question 'name' data to the document 'title' and the question 'content' data to the document 'content'. The rest of the object is pretty much boilerplate from the base_activity class code and the documentation.

The function I create looks like this (full file here):

    public function get_document($record, $options = []) {
        try {
            $cm = $this->get_cm('questionnaire', $record->questionnaireid,
                $record->courseid);
            $context = \context_module::instance($cm->id);
        } catch (\dml_missing_record_exception $ex) {
            // Notify it as we run here as admin, we should see everything.
            debugging('Error retrieving ' . $this->areaid . ' ' . $record->id .
                ' document, not all required data is available: ' .
                $ex->getMessage(), DEBUG_DEVELOPER);
            return false;
        } catch (\dml_exception $ex) {
            // Notify it as we run here as admin, we should see everything.
            debugging('Error retrieving ' . $this->areaid . ' ' . $record->id .
                ' document: ' . $ex->getMessage(), DEBUG_DEVELOPER);
            return false;
        }

        // Prepare associative array with data from DB.
        $doc = \core_search\document_factory::instance($record->id, $this->componentname,
            $this->areaname);
        $doc->set('title', content_to_text($record->name, false));
        $doc->set('content', content_to_text($record->content, $record->introformat));
        $doc->set('contextid', $context->id);
        $doc->set('courseid', $record->courseid);
        $doc->set('owneruserid', \core_search\manager::NO_OWNER_ID);
        $doc->set('modified', $record->timemodified);

        return $doc;
    }

Prior to constructing the returned document object, the search API needs to verify that the requesting user has the right permissions to access the Moodle item. This is done with check_access() function. Generally speaking, what this function should do is to extract the Moodle item in the correct Moodle context, and then check that item and context with the requesting user's capabilities. In my case, a question's data is tied to the questionnaire it belongs to. So determining if a user can see that is based on their access to the specific questionnaire instance.

For questionnaire, this is complicated by the fact that a survey (which contains the questions) can belong to more than one questionnaire (in the case of public questionnaires). While this is an uncommon use, it can still exist. The check_access() function only provides me with the unique id of the data that was saved. For questions, this is the question id field. This means that I do not know which questionnaire instance I need to check when I receive this id data. So my function will need to "guess" at which questionnaire to return by looking for one that contains the question data that is accessible to the user.

The function I create looks like this (full file here):

    public function check_access($id) {
        global $DB;

        try {
            // Questions are in surveys and surveys can be used in multiple questionnaires.
            $question = $DB->get_record('questionnaire_question', ['id' => $id],
                '*', MUST_EXIST);
            $questionnaires = $DB->get_records('questionnaire',
                ['sid' => $question->survey_id]);
            if (empty($questionnaires)) {
                return \core_search\manager::ACCESS_DELETED;
            }
            $cmsinfo = [];
            foreach ($questionnaires as $questionnaire) {
                $cmsinfo[] = $this->get_cm('questionnaire', $questionnaire->id,
                    $questionnaire->course);
            }
        } catch (\dml_missing_record_exception $ex) {
            return \core_search\manager::ACCESS_DELETED;
        } catch (\dml_exception $ex) {
            return \core_search\manager::ACCESS_DENIED;
        }

        // Recheck uservisible although it should have already been checked in core_search.
        // Use the first one that is visible. Otherwise exit.
        $cmvisible = false;
        foreach ($cmsinfo as $cminfo) {
            if ($cminfo->uservisible !== false) {
                $cmvisible = $cminfo;
                break;
            }
        }
        if ($cmvisible === false) {
            return \core_search\manager::ACCESS_DENIED;
        }

        $context = \context_module::instance($cminfo->id);

        if (!has_capability('mod/questionnaire:view', $context)) {
            return \core_search\manager::ACCESS_DENIED;
        }

        return \core_search\manager::ACCESS_GRANTED;
    }

Without going into too much detail, the basic function is:
  • find the question data and all questionnaires that might contain it,
  • get a valid course module for each questionnaire,
  • find the first course module that is visible to the user and assume this is the one,
  • check that the user has the proper capabilities to view that questionnaire.
For this post, the functionality will suffice to show how this function works. Unless the function finds reason not to grant access, then granted access is returned.

(Note - I may be able to modify my functions to return a search id field that I construct to identify both the question and the questionnaire. I will look into this later, as I am not happy with "guessing" which questionnaire is used.)

The last two functions I need to provide are get_doc_url() and get_context_url(). These functions provide the link to the resulting page that will display the found content in its proper Moodle context. Since there is no way to look at only a question instance, my functions will provide a link to the questionnaire. Both functions will provide the same link. I write the code for the get_context_url() and just call that from the get_doc_url() function.

The get_context_url code looks like this (full file here):

    public function get_context_url(\core_search\document $doc) {
        $context = \context::instance_by_id($doc->get('contextid'));
        return new \moodle_url('/mod/questionnaire/view.php',
            ['id' => $context->instanceid]);
    }

Pretty straightforward.

Since I have everything I need, I load the new code, reindex all of the site contents and give it a test.

One of the first things I notice is that there is a new "search area" in the dropdown called "Questionnaire - questions":


Next, I search for a term that I know is only in some questions. Sure enough, I see the question name and the question content where matches occur. And clicking the link takes me to the appropriate questionnaire.

Updating your plugin to use the new global search feature is a fairly easy process, especially if all you do is provide the basic search. There should be no reason not to do it!

This concludes my series on adding global search to your Moodle plugin. Feel free to leave comments and suggestions. I will look at improving the question results as I noted above, and will provide updates when I have completed that.

Checklist for building additional plugin content searching:
  1. Create a new class with the name of your area in a similarly named file in your 'classes/search' directory.
  2. Add a language string name using "search:AREANAME" for the search area in your plugin language file.
  3. Create a get_recordset_by_timestamp() function to retrieve the records you want to index.
  4. Create a get_document() function to return a completed document object to the search function.
  5. Create a check_access() function to determine a matched document's availability to the user performing the search.
  6. Create a get_context_url() function to return an appropriate Moodle URL to the matched document.
  7. Create a get_doc_url() function to return an appropriate Moodle URL to the matched document.

Update - 2017.02.02
After a lot of discussion with other developers, I redid the activity search to include the question content. Since the question content was helping to find the questionnaire instance that contained it, it really made sense for it to be a part of the activity search and instance return. Since there could be many question records per questionnaire instance, I had to do the database search in the get_document() function and add the question data there. You can see the resulting work here.


Thursday, January 26, 2017

Adding search to your Moodle plugin - part two

Part Two - Indexing Custom Information

In the last post, I created the basic code required to allow Moodle's global search feature search my plugin's "name" and "intro" information fields. Now I want to extend that functionality to include the "subtitle" and "additional info" fields.

Reading the documentation, this looks like it should be an extension of the functionality I wrote for the basic case. The information fields I want to return, both belong to an activity instance. And the case I already created is for that activity instance. So I want to look at extending the class I already created.

If I decide to provide index data from other sources, then I would be creating new classes as described in the documentation following sections.

One of the concerns I have with my approach of continuing extending the base_activity class, rather than the base_mod class, is that the documentation says that I should use base_activity "to index Moodle activities basic data like the activity name and description". And that base_mod should be used for "other specific activity data". The two extra fields I want to include, aren't part of the main activity table of my plugin, but rather are stored in a second table called "questionnaire_survey". There is a one-to-one relationship though, and the information is considered part of the activity information, so I still think this the correct approach. But, to get that extra data, I will have to override the database function that gets all of that data.

Looking at the /search/classes/base_activity.php file, I see that the main function to retrieve and return the activity data is the get_recordset_by_timestamp function:

    public function get_recordset_by_timestamp($modifiedfrom = 0) {
        global $DB;
        return $DB->get_recordset_select($this->get_module_name(),
                   static::MODIFIED_FIELD_NAME . ' >= ?',
                   array($modifiedfrom),
                   static::MODIFIED_FIELD_NAME . ' ASC');
    }

This is the one I will override in my class. In its current form, it returns all of the fields from the main activity table according to the "timemodified" field. Since I want to JOIN the "questionnaire_survey" with this table, I am going to have to be selective about the fields that are returned, as there is overlap.

I open my questionnaire/classes/search/activity.php file and add the following function (full file here):

    public function get_recordset_by_timestamp($modifiedfrom = 0) {
        global $DB;

        $sql = 'SELECT q.*, s.subtitle, s.info ' .
            'FROM {questionnaire} q ' .
            'INNER JOIN {questionnaire_survey} s ON q.sid = s.id ' .
            'WHERE q.timemodified >= ? ' .
            'ORDER BY q.timemodified ASC';

        return $DB->get_recordset_sql($sql, [$modifiedfrom]);
    }

Since I haven't told the search engine anywhere that I want to include the two new fields I've added, there must still be some changes to make. I install the new code, reindex the global search and test that things work. While there are no failures, I also cannot get any matches from the "subtitle" or "additional info" fields, confirming that there is more to do.

Looking at the documentation again, the next function that is presented is get_document. From the description, this function takes a record from the query results of get_recordset_by_timestamp function, and constructs a \core_search\document object with all of the data needed for indexing. My guess, is that I need to add my two new fields here. I'm going to override this function in my class.

The main function returns a constructed object with the data added to it as variables. I believe I can override this and take the returned function from the parent, add my new data to it, and return it. When I'm done, the function looks like this:

    public function get_document($record, $options = []) {
        // Get the default implementation.
        $doc = parent::get_document($record, $options);

        // Add the subtitle and additional info fields.
        $doc->set('subtitle', content_to_text($record->subtitle, false));
        $doc->set('additionalinfo', content_to_text($record->info, $record->introformat));

        return $doc;
    }

I add the code to my site, purge all the caches and reindex the search engine. But, searching for these fields is not returning my data. On the search areas admin page, I have been using "Update indexed contents". But, since none of my questionnaire activities have actually changed, nothing is being re-indexed. Instead, I need to use the "Reindex all site contents" button. When I do this, I get an error indicating:

    Coding error detected, it must be fixed by a programmer: "subtitle" field does not exist.

It would appear that using "subtitle" is not correct.

The documentation shows an example of using get_document(), and it uses "description1" and "description2" as the fields being set. Its likely that I can't use "subtitle" and "additionalinfo" as  field names in the $doc->set() function.

The \core_search\document class is defined in the /search/classes/document.php file. The set() function uses a static array to look for specific field names it is allowed to set. The names I used, "subtitle" and "additionalinfo" are not part of those. However "description1" and "description2" are. This confirms that I need to change those functions. The set() function throws an exception when an unknown field is used, and that explains the error I saw.

My new function now looks like this (full file here):

    public function get_document($record, $options = []) {
        // Get the default implementation.
        $doc = parent::get_document($record, $options);

        // Add the subtitle and additional info fields.
        $doc->set('description1', content_to_text($record->subtitle, false));
        $doc->set('description2', content_to_text($record->info, $record->introformat));

        return $doc;
    }

When I load it, and reindex all of the site contents, my search works. I now see content found from a subtitle.


So it seems that I can add indexed fields to the a activity level search, but only two. And they need to be referred to as "description1" and "description2". I'm guessing to add more than that, I would need to create more classes using base_mod instead of base_activity.

I'll explore that more in the next post, as well as how to search for more parts of my plugin.

Monday, January 23, 2017

Adding search to your Moodle plugin

Part One - The Basics

In Moodle 3.1, the "Global Search" feature was introduced. This feature provides the ability for a user to search for text within the entire site for entered terms. The results obey all role and capability rules, so only parts that can be legitimately accessed by the requesting user are returned.

As delivered, core activities are included in the global search. But plugins need to provide code to the search API in order to be included in the search results. In this series of blog posts, I am going to add the search feature to my questionnaire plugin.

Some resources that will help with this task are:


The global search feature uses search engine plugins to provide the search functions. Currently, there is only one plugin, that uses an external technology called Solr. In order to use search, and enable me to test my work, I will install it in my development environment. Moodle provides documentation to do this. Since I am using OSX with Macports, I follow those instructions. After that, I use the administration functions in my Moodle site to enable global search and verify that it is working. I'm not going to go into the details of that here, but the documentation links were enough for me to successfully do this.

Before I start, I'm going to use the search function to see how it works with the existing activities, and verify that it is not working for mine. I'm using Moodle 3.2 with the Boost theme. The search function is located in the upper right part of the navigation bar, with a text box entry next to a magnifying glass icon as below:


Once I enter my search text, for example "test", the screen displays everything it found and some further options. Under the "filter" section, there is a "Search area" drop down that lists all of the areas available to search within (see image below). My plugin does not appear there, so I presume this will be one of the places that I will see a change once I have enabled search in my plugin.


I also notice that in those filter choices, some of the activities show only "activity information" while others show this and other levels of the activity. These correspond to what content an activity returns to the search function. Looking at the documentation, I see that returning the "activity information" is the easiest case, and does not require much work at all. I am going to start there.

I have created a working branch off of my latest Moodle 3.2 branch called GLOBALSEARCH_32 as my starting point. You can follow along on Github here.

The documentation makes this look pretty straight forward. I create a subdirectory called search in my plugin's classes subdirectory. In that new directory, I add a file named activity.php. The contents, are basically this (you can see the entire file here):

    namespace mod_questionnaire\search;

    class activity extends \core_search\base_activity {
    }

I'm going to update my development site with this new code, and see what happens. After copying, I check the filter dropdown as above, but I don't see the questionnaire in the list. In all likelihood, I need to purge all of the caches. After doing that, the page reloads, but gives me the following error:

Invalid get_string() identifier: 'search:activity' or component 'mod_questionnaire'. Perhaps you are missing $string['search:activity'] = ''; in mod/questionnaire/lang/en/questionnaire.php?
  • line 349 of /lib/classes/string_manager_standard.php: call to debugging()
  • line 7076 of /lib/moodlelib.php: call to core_string_manager_standard->get_string()
  • line 146 of /search/classes/base.php: call to get_string()
  • line 62 of /search/classes/output/form/search.php: call to core_search\base->get_visible_name()
  • line 204 of /lib/formslib.php: call to core_search\output\form\search->definition()
  • line 59 of /search/index.php: call to moodleform->__construct()
It looks like I need to define a specific language string as well. The error nicely tells me exactly what I need. I add the line:
    $string['search:activity'] = 'Questionnaire - activity information';
to the lang/en/questionnaire.php file and try again. This time, I have no error, and the drop down shows my activity in the list:


However, when I perform a search that should include a questionnaire, it does not show up. The most likely reason is that the search engine has not indexed the plugin into its database yet. The global search can be scheduled to reindex its database, and it can also be manually reindexed. At the main global search administration page, admin/settings.php?section=manageglobalsearch, there is an "Index data" link. Reviewing that page, I can see that there is a section for "Questionnaire - activity information", that is showing as never having been indexed. When I click the "Update indexed contents" button, also on that page, my questionnaire now shows as having been indexed. I perform my search again, and I can now see results from my questionnaire.

Adding the basic search functionality turned out to be extremely easy. I simple had to add one file that extended a class, and one language string. In the next post, I'll add extra levels of searching to include questionnaire "subtitle" and "additional info" sections.


Tuesday, January 17, 2017

Part 5 - Further discussion

After I completed my first attempt at adding renderers and templates to my questionnaire plugin, I went on to use them much more thoroughly in the actual release. The Moodle 3.2 release of questionnaire uses templates for all of the major pages. This is a work in progress, and I have taken some shortcuts to make the transition easier while allowing me to release the code as it is worked on.

Most noticeably, since the output of questionnaire is generated by specific functions that return HTML pieces, I have created the templates to accept that HTML. This means that only the top level formatting of a page is templatable. But, I am working on extracting more and more of the output into the templates as the work continues.

In the RENDERER_32 work in progress branch, I have refactored and extracted almost all of the question specific output into individual question type templates. These templates are used with renderer functions to return the HTML to the main renderer. I have coded the renderer function so that it will use question templates as I define them, or fallback to the old method if they are not present. If you have questions about this work, contact me or post a question here.

If you wander through those templates, you'll see some other aspects of templates used that I didn't discuss in this series. For example, in the navbaralpha.mustache file you can see the use of the language string {{# str}} helper function. This is a way to put a language string directly into the template. You can find more information on that here.

In the question_check.mustache template, you can see the use of dotted notation for tag names. For example, choice.id. While not strictly necessary, I find that this notation helps to document the structure of the context data being used in the template. In this example, the fact that id is a subelement of the choice structure.

Regarding data structures, if you recall in "Part three", I mentioned that the template data could be formatted as arrays or objects and that both would work. The code I provided used strictly arrays. But it could have used objects as well as the one array needed for the rows element. Below is an alternate coding using objects instead of arrays where possible:

    /**
     * Prepare data for use in a template
     *
     * @param \renderer_base $output
     * @return stdClass
     */
    public function export_for_template(\renderer_base $output) {
        $data = new \stdClass();
        $data->headings = new \stdClass();
        foreach ($this->headings as $key => $heading) {
            $data->headings->{$key} = $heading;
        }
        $data->rows = [];
        foreach ($this->rows as $row) {
            list($topic, $name, $responses, $type) = $row;
            $item = new \stdClass();
            $item->topic = $topic;
            $item->name = $name;
            $item->responses = $responses;
            $item->type = $type;
            $data->rows[] = $item;
        }
        return $data;
    }

Lastly, I never got into using javascript with templates. I have personally not spent enough time with this to do that topic justice. If you wish to explore that though, start with the documentation. Additionally Damyon Wiese from Moodle HQ gives a great demonstration that covers how to include javascript and Ajax with templates, as well as a great introduction to templates in general. If you want to skip ahead to where javascript comes into play, fast-forward to around 14:10 in the video.

If you want to discuss any of this series, or recommend alternate approaches, please feel free to leave comments or contact me directly.