Friday, May 19, 2017

Things I learned upgrading to Moodle 3.3

As the Moodle 3.3 launch approached, I decided to check on a few of my plugins and ensure they were ready for 3.3.

The good news is that having had everything pretty much up to 3.2 standards, the effort to reach 3.3 was minimal. In fact, with the two plugins I worked on for the coveted "Early bird 3.3" award, questionnaire module and oembed filter, I could have submitted them as is and they would have passed fine. But, that's not my way, and besides, it would have made this post even shorter than it is.

For the questionnaire module, I first installed the 3.2 version on a new Moodle 3.3 test site, and ran the unit tests and Behat tests. Having automated testing is a great way to manage upgrading, as these tests will hopefully uncover any problems introduced by the Moodle upgrade or by any changes made to the plugin for the new version. In the questionnaire case, I had made a number of improvements to the code, continuing my efforts of paying down "technical debt" and making the code easier to maintain going forward.

The unit tests ran fine, but the Behat tests failed. Additionally, there were messages being kicked out by the developer level debugging.

There were two issues in Behat  I needed to fix. The first is a change to the Behat tests in Moodle 3.3. Where previously a step would say:
'And I follow "Course 1"' 
For 3.3, they need to say:
'And I am on "Course 1" course homepage'
This is due to the new course dashboard introduced in 3.3 that replaces some of the navigation in 3.2.

The other issue I needed to change was places that would say:
'I navigate to "Questions" node in "Questionnaire administration"'
Now need to say:
'I navigate to "Questions" in current page administration'
This was actually a change in 3.2 that didn't affect me until 3.3. I believe I was supposed to have made the change in 3.2, but now it is mandatory.

Making those changes fixed the Behat errors.

The debugging warning I was getting was:
pix_url is deprecated. Use image_url for images and pix_icon for icons.
    line 267 of /lib/outputrenderers.php: call to debugging()
    line 182 of /mod/questionnaire/classes/questions_form.php: call to renderer_base->pix_url()
    line 204 of /lib/formslib.php: call to mod_questionnaire_questions_form->definition()
    line 32 of /mod/questionnaire/classes/questions_form.php: call to moodleform->__construct()
    line 153 of /mod/questionnaire/questions.php: call to mod_questionnaire_questions_form->__construct()
This is telling me that places I am using the function pix_url will eventually (in some future release) fail. And they should be replaced with either pix_icon or image_url. For my case, image_url is the easiest and correct fix. The previous function, pix_url, returned an URL to be used in the output. The new function, image_url, likewise returns an URL. The new function pix_icon returns the entire line of code including the image tags. Using it would require a significant amount or re-coding, and in reality, I am not using icons.

Once I have completed those changes, everything seems fine with questionnaire.

For the Oembed filter, I ran into Behat issues caused by the HTML that is output with 3.3. There has been changes made to some of the core editing features, that changed the xpath layout for the Oembed editing page. As such, I was getting an error like:
001 Scenario: Admin user carries out various provider management tasks. # /Users/mikechurchward/www/moodlehq.git/filter/oembed/tests/behat/management.feature:26
      And the provider "Vimeo" is disabled                              # /Users/mikechurchward/www/moodlehq.git/filter/oembed/tests/behat/management.feature:36
        The "//td/a[text()='Vimeo']/parent::td/div/a[contains(@class,'filter-oembed-visibility')]/i[@title='Show']" element does not exist and should exist (Behat\Mink\Exception\ExpectationException)
The Behat tests for the filter had a custom test called "the_provider_is_disabled" which depended on a specific xpath output that was now no longer happening. This required me to rewrite the Behat function to change the xpath definition to reflect the new output. Manual testing of the function proved that the actual functionality had not been broken, and once the new xpath definition was in place, the Behat tests passed as well.

And that was it. Both plugins were ready for 3.3 and made it in time to get the early bird award. As I continue with other plugins, I will document any new changes required by 3.3 I find in new posts.

Friday, April 21, 2017

Add mobile support to your Moodle plugin - part four

Part Four - Adding More Services

In the previous part of this series, I modified my mobile addon so that clicking on a plugin instance loaded and displayed a "Hello World!" screen. Now, I will add some services so that it will retrieve the number of responses made to the particular questionnaire and display them, when a user clicks on an instance. I'm also going to add a link handler, that will handle the questionnaire when it is selected by a link in other site content.

First, I'm going to add the link function. In all of the other addons I've looked at, a handler was defined for links. In the moodle.org forum, I asked about this, and learned that it was so the mobile plugin was handled properly if it was reached from a link other than the course instance.

Adding the link function is apparently very easy now. The latest release of the mobile app has a simplified handler function that manages the links for me. I really just have to add one line to my main.js file and one line to my services/handlers.js file, as follows:

main.js:
.config(function($mmCourseDelegateProvider, $mmContentLinksDelegateProvider) {    $mmCourseDelegateProvider.registerContentHandler( 'mmaModQuestionnaire', 'questionnaire', '$mmaModQuestionnaireHandlers.courseContent');    $mmContentLinksDelegateProvider.registerLinkHandler( 'mmaModQuestionnaire', '$mmaModQuestionnaireHandlers.linksHandler');
services/handlers.js:
self.linksHandler = $mmContentLinksHelper.createModuleIndexLinkHandler( 'mmaModQuestionnaire', 'questionnaire', $mmaModQuestionnaire);
Now, when I create a link in my course that goes to a questionnaire instance, instead of launching in the browser, it uses the questionnaire mobile plugin.

In the last iteration, I left the isPluginEnabled function in a state where it always returned true. Now, I need to make it do something that actually checks with the site to see if the plugin really is enabled. This will require changing this function in the mobile plugin, and adding a web service to the questionnaire module.

Both the certificate and feedback call the 'mod_[pluginname]_get_[pluginname]_by_courses' web service, so I'll start there. Note that feedback also calls mod_feedback_get_feedback service, but I'll look at that later.

Starting with the mobile plugin, I'll update the services/questionnaire.js::isPluginEnabled function as follows:
self.isPluginEnabled = function(siteId) {
    siteId = siteId || $mmSite.getId();
 
    return $mmSitesManager.getSite(siteId).then(function(site) {
        return site.wsAvailable( 'mod_questionnaire_get_questionnaires_by_courses');
    });
};
What this code does, is verify that the questionnaire module is enabled on the Moodle site by checking for the availability of the web service, mod_quesrionnaire_get_questionnaire_by_courses, on the Moodle site. This means I will need to create that web service in the main questionnaire activity plugin. For this one, I am going to copy from the certificate module, but simplify it down for now to only provide the basic module information and only check for 'view' capabilities.

To add the web service to my module, I do the following:

Add the necessary external services to the classes/external.php file:

This requires creating the file with the external class extending the core external_api class in the mod_questionnaire namespace. In that class, I need to add the function for get_questionnaires_by_courses as well as two other helper functions, get_questionnaires_by_courses_parameters and get_questionnaires_by_courses_returns. These three functions are required to fully define the web service as described in the web services developer documentation.

It is the job of this function to return result information for all questionnaire instances in the identified courses. You can see the full code for this here.

Add the db/services.php file to define the services available by web services:

Web services need to be described in this file. When a plugin is installed or upgraded, the Moodle system uses this file to add the services described in this file to its known, available services. This is more fully described here. Once the Moodle site knows about them, it can allow them to be used by external systems.

Since I already have the questionnaire module installed on my Moodle site, I will need to bump up the version number in my version.php file, to trigger a module upgrade, and get the Moodle site to load my new web service. Once I have done this, I should be able to see that my new service has been installed by going to my site's admin/webservice/service_functions.php screen and looking for the web service there. I upgrade my site with new version and check for the service on the admin screen. Successfully, I see the following:


Since I'm writing web services, I might as well create one that provides some meaningful information from the Moodle site that I can display in my mobile plugin.

Currently, when I click on a questionnaire in the mobile app, I just get the "Hello World!" screen. I am going to modify it so that it tells me how many responses I have made to that instance. To do that, I will need a web service in the Moodle plugin to provide that information back to the mobile plugin.

Back at my db/services.php file, I define a new service called 'mod_questionnaire_get_user_responses' and define it as a service that returns 'a count of the current user responses'.

Then, I code the new service in my classes/external.php file. Its a simple function, that simply returns the count of responses for the user / questionnaire instance in question.

Lastly, I perform a version bump to force an upgrade and verify that the service has been added to the Moodle site:


Now, I need to modify the mobile plugin to use this new service and display the information appropriately. Starting with templates/index.html file, I modify so that it will display a count of the user's responses:
<ion-view>
    <ion-nav-title>{{ title }}</ion-nav-title>
    <ion-content padding="true" mm-state-class>
        <mm-course-mod-description description="description"></mm-course-mod-description>
        {{mymessage}}<br />
        You have {{responses}} responses.
    </ion-content>
</ion-view>
The template will look for a value in the {{responses}} tag, which means I need to add code to the controllers/index.js file to populate that. The code includes a function that uses the promise mechanism to add the response count to the template scope variable, and a helper function to call the plugin service function, $mmaModQuestionnaire.getUserResponses located in services/questionnaire.js.

In the services/questionnaire.js file, I add the code that will call the new web service I created in the Moodle plugin. This is the part that will actually call the site service and extract the response information. The actual call to the web service is in this line:
return $mmSite.read('mod_questionnaire_get_user_responses', params, preSets).then(function(response) {
This should be all I need to change the behaviour of the questionnaire module in the mobile app, so that it will also show me the number of responses.

With my new code added to the mobile plugin, and the new services added to the Moodle module, I need to verify that I can still access the questionnaire in the mobile app, and that I can see the number of responses.

I ramp up my mobile app, click the questionnaire instance, and achieve success:


I still have a lot of work ahead of me to make my questionnaire plugin functional in the mobile app, but I at least now have a good working understanding of what needs to be done, how to use web services, and how to make them useful in the mobile plugin. I will get back to this (eventually), and document my learnings in future posts.

Friday, March 31, 2017

Add mobile support to your Moodle plugin - part three

Part Three - Starting to Build the Moodle Mobile Add-on

In the previous part of this series, I created just enough code to see an impact on the mobile app. Now, I want to begin to actually do things that will lead me to a solution.

To start with, I decided to take an introductory lesson in AngularJS. If you are not familiar, I highly recommend Code School's Shaping up with AngularJS. This helped me to understand a lot of the javascript I am looking at with other mobile add-ons.

Learning this little bit, has helped me to understand the code I am looking at, and helps me to make good guesses about what is going on with the Moodle specific code.

I'm going to use some existing plugins as a starting point for the code I need. Specifically, the certificate module, which is a third party activity plugin, and the feedback module, which will be included in core.

The main.js file defines the necessary parts to the mobile app. The stateProvider configuration sets up the module for displaying when it is clicked in the course. In the certificate code, you can see a controller and a template defined to do this:
controller: 'mmaModCertificateIndexCtrl',templateUrl: 'addons/mod/certificate/templates/index.html'
In AngularJS, this indicates that the display setup code will be in controllers/index.js and that will use templates/index.html to display that code. When a user clicks on a questionnaire instance, these will be the main pieces to determine what is displayed.

The other part of the main.js file, registers the various handlers needed by the mobile app. In the certificate example, you can see there are two: a "course delegate provider" and a "links delegate provider". The course delegate provider is the part responsible to provide functionality for the module to that mobile app. Its job is to handle a click, and use the module plugin code. Without this, the mobile app will display the "not available" message. The links delegate provider provides function for other links to a module instance, not from the main course display. For example, if a link is put into a forum post. Without this, a link will simply launch a web link in the default browser.

For my module, I'm going to focus on the course delegate provider first. And I will use the index files to affect what is displayed in the mobile app.

For this part of the discussion, I have posted my files here. This is the minimum I have determined I need to have my plugin work in the mobile app without any errors.

To start with, I have created my main.js file to define that the questionnaire plugin should be available, that it uses an index controller and that it has a course content handler. I have taken what I started in the last post, and extended out to provide what is absolutely required.

Next, I will further flesh out my services/handlers.js file. In the first iteration, I only provided an isEnabled function, and it simply returned true. Now I have improved that to call a specific function provided by my questionnaire handler:
self.isEnabled = function() {
    return $mmaModQuestionnaire.isPluginEnabled();
};
That function will eventually call web services from the Moodle site to verify that the module is enabled.

I have also added a getController function, which is required by the framework to provide information for displaying and launching my plugin code, as follows:
self.getController = function(module, courseid) {
    return function($scope) {
        $scope.title = module.name;
        $scope.icon = 'addons/mod/questionnaire/icon.svg'
        $scope.class = 'mma-mod_questionnaire-handler';
        $scope.action = function() {
            $state.go('site.mod_questionnaire', {module: module, courseid: courseid});
        };
    };
};
This code tells the framework that my main handler class will be located in services/questionnaire.js and will be called mmaModQuestionnaire. I have also added an icon to be displayed on the course page, and indicated that it will be located in the root of my plugin code. I copied that icon from the main questionnaire plugin. When I run the mobile app with this code, I should now see the questionnaire icon, rather than the default icon, on the course page.

The changes I have made here, mean that I need to create a services/questionnaire.js file, with an mmaModQuestionnaire class, that provides an isPluginEnabled method. You can see that file here. For now, I am keeping the function simple:
self.isPluginEnabled = function(siteId) {
    siteId = siteId || $mmSite.getId();
    return Promise.resolve(true);
};
The function returns a Javascript Promise (if, like me, you aren't familiar with that, this helped me). Essentially, somewhere in the framework, there is code that will call isEnabled in my course handler and will expect a Promise construct. Since I have not written the necessary web services on the other end to verify that, I am simply going to return a resolved promise to indicate my questionnaire plugin should work.

The last parts I need to create are the index display handlers I described in the main.js file. I have created the controllers/index.js file here, and the templates/index.html file here.
As I mentioned previously, these files work together to define and display what is output when a user clicks on a questionnaire instance in the mobile app. For now, I will set it up to simple display "Hello world!".

The key elements in index.js are the properties of the $scope variable:
$scope.title = module.name;
$scope.description = module.description;
$scope.courseid = courseId;
$scope.mymessage = "Hello World!";
These are the bits that I can use as variables in the index.html file to display specific information. In Angular, this means parts contained in "{{}}" or defined as tag elements. The template I create looks like this:
<ion-view>
    <ion-nav-title>{{ title }}</ion-nav-title>
    <ion-content padding="true" mm-state-class>
        <mm-course-mod-description description="description"></mm-course-mod-description>
        {{mymessage}}
    </ion-content>
</ion-view>
The {{ title }}, <mm-course-mod-description description="description"> and the {{mymessage}} elements should be replaced by the corresponding $scope element.

All of this should be enough for me to reload my mobile app questionnaire plugin code into the mobile app, run it and when I click on a questionnaire link, see a page that says "Hello world!". So I copy my code into the app www/addons/mod/questionnaire directory, run the server and check out my app.

The course page now displays this:



Note, that my questionnaire icon is now displaying, rather than the default puzzle piece. Clicking on the questionnaire instance, shows me this:


Which shows me the title and my "Hello World!" message. Looks like I have succeeded.

I'm going to leave it there for this post. In my next post, I will build some services to interact with, and build out the link delegate handler as well.


Tuesday, March 21, 2017

Add mobile support to your Moodle plugin - part two

Part Two - Exploring the Moodle Mobile Add-on

In the first part of this series, I set up a development environment on my local machine. I am going to use this same environment to begin coding the mobile add-on portion of my Moodle questionnaire plug-in.

The documentation says I need to develop the mobile side as a standard Moodle Mobile add-on and then package it as a remote add-on, with four steps:
  1. Develop the required Moodle Web Services
  2. Develop a standard Moodle Mobile add-on
  3. Package the Moodle Mobile add-on as a remote add-on
  4. Include the remote add-on in your Moodle plugin
The first two steps are where my work will start. The second two steps are what I will need when I am ready to distribute what I have developed for anyone to use in their mobile apps with their Moodle sites. I am also going to start with step two, as this will help me understand what is needed in the mobile portion and what is needed from the Moodle plugin portion.

Recall, that when I try to access my plugin in the mobile app now, I get the screen:


My goal, for the start, is to provide enough code so that screen changes.

To start, I look at the current structure of the mobile app on my local server. Within it is a www/addons directory that contains all of the core mobile addons. I can see, that like a Moodle site, there is a mod subdirectory, and within that, a number of subdirectories corresponding to each of the supported core activity plugins.

For ease of development, I am going to add my plugin to the same location. That way, I can update my code within the mobile app, and access it with the browser, to see my work. Essentially, my local files will stand-in as the mobile app, and the questionnaire mobile addon will act as a core activity plugin.

For purposes of code storage, I am going to add an addons subdirectory to my questionnaire repository and add the code that will go into the mobile addon there. This code will not be required, nor run, on a Moodle site, but I want to keep it together with my Moodle plugin for development ease.

Through a productive discussion with the mobile development team, I have determined that getting my add-on registered with the mobile app requires code that registers a content handler. In the add-on structure, this happens in the main.js file in a config statement (full code here):
.config(function($mmCourseDelegateProvider) {
    $mmCourseDelegateProvider.registerContentHandler('mmaModQuestionnaire', 'questionnaire', '$mmaModQuestionnaireHandlers.courseContent');
});
Essentially, the mobile framework translates this code such that it knows there must be a services subdirectory and within that a file named handlers.js that contains a courseContent function. I create that structure with a very simple courseContent function as follows (full code here):
self.courseContent = function() {
    var self = {};
    /**
     * Whether or not the module is enabled for the site.
     *
     * @return {Boolean}
     */
    self.isEnabled = function() {
        return true;
    };
    return self;
};
In conversations with the developers, I have determined that I need an isEnabled function, that returns true if the plugin is enabled, for my addon to be acknowledged and not display the "plugin that is not yet supported" message. For now, I just send true with no logic confirming that. Eventually, this will call a web service from the actual questionnaire plugin on the Moodle site to determine this. I also know that when I run my local version of the mobile app using the ionic server, it will build my addons code into the app's main code file, www/build/mm.bundle.js.

After dropping these two files into my mobile app's www/addons/mod/questionnaire directory, I run the ionic server (removed some messages for brevity):
ionic serve --browser chromium
Running 'serve:before' gulp task before serve
[13:43:32] Starting 'build'...
[13:43:32] Starting 'sass-build'...
[13:43:32] Starting 'lang'...
[13:43:32] Starting 'watch'...
[13:43:41] Finished 'watch' after 9.04 s
[13:43:47] Finished 'sass-build' after 15 s
[13:43:47] Starting 'sass'...
[13:43:48] Finished 'lang' after 15 s
[13:43:49] Finished 'sass' after 1.92 s
[13:43:50] Finished 'build' after 18 s
[13:43:50] Starting 'config'...
[13:43:50] Finished 'config' after 17 ms
[13:43:50] Starting 'default'...
[13:43:50] Finished 'default' after 6.02 μs
[13:43:50] Starting 'serve:before'...
[13:43:50] Finished 'serve:before' after 3.15 μs
Running live reload server: http://localhost:35729
Watching: www/**/*.html, www/build/**/*, www/index.html, !www/lib/**/*
√ Running dev server:  http://localhost:8100
Ionic server commands, enter:
  restart or r to restart the client app from the root
  goto or g and a url to have the app navigate to the given url
  consolelogs or c to enable/disable console log output
  serverlogs or s to enable/disable server log output
  quit or q to shutdown the server and exit
ionic $
I can see a lot of build activity happening. I then point my browser to the server instance and see a lot more activity kick in. When I access my course, and then click on my questionnaire activity, I no longer get the "plugin that is not yet supported" message:


I don't get the questionnaire displayed, but I didn't expect to. I have at least learned how to impact the mobile app with my addon.

I also take a look at the www/build/mm.bundle.js to see if it now contains the code I created for the questionnaire addon. A quick search of the code reveals that it does. So I have confirmed that the mobile app has successfully picked up the questionnaire addon code and added it to the app.

In the next post, I will flesh out more of the mobile add-on code, and begin to look what the web services needed on the questionnaire plugin side.

Friday, March 3, 2017

Add mobile support to your Moodle plugin - part one

Part One - Setting up a Development Environment

With version 3 of Moodle came the Moodle Mobile app. Out of the box, most of the core Moodle plugins came supported. But, third-party plugins do not function by default in the mobile app.

In Moodle 3.1, the mobile app added support for "Remote add-ons". This support allows plugin providers to add support so that their plugin can function within the mobile app.

Now, I should point out that Moodle is currently working on a much simpler system for adding mobile support to your plugin, but it will be a while before it is ready. When it is ready, the work I am about to undertake will likely be moot.

This post will be the first in a series where I will attempt to add mobile support for my questionnaire module. I say "attempt", because this will be using some technologies that I am not familiar with so I will be learning as I go. I won't guarantee success, but I will document the efforts.

To start with, the main developer documents for Moodle Mobile are here. Looking through that list, I'm going to start with setting up a development environment on my Mac.

I already have Chrome installed on my Mac, so I can skip that step. The next step it says is to install "Node.js". Keep in mind, I don't really know what I am doing here (yet), so I'm not sure how this comes into play, but I'm going to follow instructions. :-)

The documentation suggest using Macports to install Nodejs. It also says I need to use version "v0.12.7". But, it looks like Macports only has versions from 4.* to 7.*. So, Macports is out. The documentation also provides a direct download link for "v0.12.7", and it has a Mac "pkg". So I download that and run the Mac installer. As a result, I have "node" installed at /usr/local/bin/node and "npm" installed at /usr/local/bin/npm.

The next step asks me to run the command npm cache clean. When I do this, I get a series of errors. which seem to indicate permission problems. Likely I need to elevate the permission level this command runs at, so I add a sudo to the front of the command and try again. The new command, sudo npm cache clean works fine. I'm going to assume I will need sudo for the rest of the commands in the document as well.

Next, I am asked to install "ionic". I excute the command:
sudo npm install -g cordova ionic
I get some issues displayed:

npm WARN engine cordova@6.5.0: wanted: {"node":">=4.0.0"} (current: {"node":"0.12.7","npm":"2.11.3"})
npm WARN engine request@2.79.0: wanted: {"node":">= 4"} (current: {"node":"0.12.7","npm":"2.11.3"})
npm WARN deprecated node-uuid@1.4.7: use uuid module instead
/usr/local/bin/cordova -> /usr/local/lib/node_modules/cordova/bin/cordova
/usr/local/bin/ionic -> /usr/local/lib/node_modules/ionic/bin/ionic
cordova@6.5.0 /usr/local/lib/node_modules/cordova
├── underscore@1.7.0
├── q@1.0.1
├── nopt@3.0.1 (abbrev@1.1.0)
├── update-notifier@0.5.0 (is-npm@1.0.0, semver-diff@2.1.0, chalk@1.1.3, string-length@1.0.1, repeating@1.1.3, configstore@1.4.0, latest-version@1.0.1)
├── insight@0.8.4 (object-assign@4.1.1, uuid@3.0.1, lodash.debounce@3.1.1, async@1.5.2, chalk@1.1.3, configstore@1.4.0, os-name@1.0.3, tough-cookie@2.3.2, request@2.79.0, inquirer@0.10.1)
├── cordova-common@2.0.0 (cordova-registry-mapper@1.1.15, ansi@0.3.1, semver@5.3.0, osenv@0.1.4, underscore@1.8.3, q@1.4.1, unorm@1.4.1, shelljs@0.5.3, minimatch@3.0.3, glob@5.0.15, bplist-parser@0.1.1, elementtree@0.1.7, plist@1.2.0)
└── cordova-lib@6.5.0 (valid-identifier@0.0.1, opener@1.4.1, cordova-registry-mapper@1.1.15, properties-parser@0.2.3, nopt@3.0.6, unorm@1.3.3, shelljs@0.3.0, semver@4.3.6, glob@5.0.15, dep-graph@1.1.0, xcode@0.9.1, elementtree@0.1.6, init-package-json@1.9.4, cordova-serve@1.0.1, request@2.47.0, tar@1.0.2, cordova-fetch@1.0.2, aliasify@1.9.0, plist@1.2.0, cordova-js@4.2.1, cordova-create@1.0.2, npm@2.15.11)

ionic@2.2.1 /usr/local/lib/node_modules/ionic
└── @ionic/app-generators@0.0.3
But they seem to be warnings about outdated "node". I know we are using an older version of "node", so I believe this will not be an issue.

Now I need to install the "bower" and "gulp" node packages.

Bower:
sudo npm install -g bower
/usr/local/bin/bower -> /usr/local/lib/node_modules/bower/bin/bower
bower@1.8.0 /usr/local/lib/node_modules/bower
Gulp:

sudo npm install -g gulp
npm WARN deprecated minimatch@2.0.10: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
npm WARN deprecated minimatch@0.2.14: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
npm WARN deprecated graceful-fs@1.2.3: graceful-fs v3.0.0 and before will fail on node releases >= v7.0. Please update to graceful-fs@^4.0.0 as soon as possible. Use 'npm ls graceful-fs' to find it in the tree.
/usr/local/bin/gulp -> /usr/local/lib/node_modules/gulp/bin/gulp.js
gulp@3.9.1 /usr/local/lib/node_modules/gulp
├── interpret@1.0.1
├── pretty-hrtime@1.0.3
├── deprecated@0.0.1
├── archy@1.0.0
├── tildify@1.2.0 (os-homedir@1.0.2)
├── minimist@1.2.0
├── v8flags@2.0.11 (user-home@1.1.1)
├── chalk@1.1.3 (escape-string-regexp@1.0.5, supports-color@2.0.0, ansi-styles@2.2.1, has-ansi@2.0.0, strip-ansi@3.0.1)
├── semver@4.3.6
├── orchestrator@0.3.8 (sequencify@0.0.7, stream-consume@0.1.0, end-of-stream@0.1.5)
├── gulp-util@3.0.8 (object-assign@3.0.0, array-differ@1.0.0, lodash._reescape@3.0.0, lodash._reinterpolate@3.0.0, lodash._reevaluate@3.0.0, beeper@1.1.1, array-uniq@1.0.3, replace-ext@0.0.1, dateformat@2.0.0, has-gulplog@0.1.0, fancy-log@1.3.0, vinyl@0.5.3, gulplog@1.0.0, lodash.template@3.6.2, through2@2.0.3, multipipe@0.1.2)
├── liftoff@2.3.0 (lodash.isstring@4.0.1, lodash.isplainobject@4.0.6, rechoir@0.6.2, extend@3.0.0, flagged-respawn@0.3.2, lodash.mapvalues@4.6.0, resolve@1.3.2, fined@1.0.2, findup-sync@0.4.3)
└── vinyl-fs@0.3.14 (strip-bom@1.0.0, graceful-fs@3.0.11, vinyl@0.4.6, defaults@1.0.3, mkdirp@0.5.1, through2@0.6.5, glob-stream@3.1.18, glob-watcher@0.0.6)
Again, there are some warnings about outdated releases. But I'm just going to leave them as is for now.

The next step talks about "Push notifications for Mac", and refers me to "https://cocoapods.org/" and the Moodle tracker item MOBILE-1970. It looks like this will resolve a problem I may have later on when running iOS versions of the app. So, I execute the command:
sudo gem install cocoapods
At the end of that, I have 27 new gems installed in /Library/Ruby/Gems/2.0.0/gems/cocoapods-1.2.0/.

I already have my git clone of the moodlemobile2 repository, so I set my current directory to it. From there, I execute:
sudo npm install
There are a number of warnings and errors that are output as this executes. Most of them appear to be problems with outdated modules. I am going to disregard those for now.

Now I run the following three commands, one after the other:
sudo ionic platform add android@5.1.1
sudo ionic platform add ios@4.1.0
sudo ionic state restore
All three seem to install okay. The only warnings I receive are for outdated dependencies.

So, I execute:
sudo bower install
And I get:
bower ESUDO         Cannot be run with sudo
So, guess I won't be using sudo here.

I rerun the command without sudo, and get a long list of files being installed. All of them went into the local repo below www/lib/.

So, gulp:

gulp
[16:30:43] Using gulpfile ~/www/moodlemobile2/gulpfile.js
[16:30:43] Starting 'build'...
[16:30:43] Starting 'sass-build'...
[16:30:43] Starting 'lang'...
[16:30:48] Finished 'sass-build' after 5.38 s
[16:30:48] Starting 'sass'...
[16:30:49] Finished 'lang' after 5.59 s
[16:30:50] Finished 'sass' after 1.7 s
[16:30:50] Finished 'build' after 7.46 s
[16:30:50] Starting 'config'...
[16:30:51] Finished 'config' after 16 ms
[16:30:51] Starting 'default'...
[16:30:51] Finished 'default' after 5.49 μs
Which again added files below my local repo.

At this point, I think I have everything ready to be able to access the Moodle Mobile app from my local development environment. The links tell me to start Chrome using:

    open -a "Google Chrome" --args --allow-file-access-from-files --disable-web-security --user-data-dir


That seems to just move my Chrome window to the front. Probably because I already had it open? I'll assume it's all good.

Now, to run the ionic server:
ionic serve --browser chromium
******************************************************
 Dependency warning - for the CLI to run correctly,    
 it is highly recommended to install/upgrade the following:  
 Please install your Cordova CLI to version  >=4.2.0 `npm install -g cordova`
 Install ios-sim to deploy iOS applications.`npm install -g ios-sim` (may require sudo)
 Install ios-deploy to deploy iOS applications to devices.  `npm install -g ios-deploy` (may require sudo)
******************************************************
WARN: ionic.project has been renamed to ionic.config.json, please rename it.
WARN: ionic.project has been renamed to ionic.config.json, please rename it.
Multiple addresses available.
Please select which address to use by entering its number from the list below:
 1) 10.120.211.137 (en4)
 2) 10.120.211.40 (en0)
 3) localhost
Address Selection: 3
Selected address: localhost
Running live reload server: http://localhost:35729
Watching: www/**/*, !www/lib/**/*
√ Running dev server:  http://localhost:8100
Ionic server commands, enter:
  restart or r to restart the client app from the root
  goto or g and a url to have the app navigate to the given url
  consolelogs or c to enable/disable console log output
  serverlogs or s to enable/disable server log output
  quit or q to shutdown the server and exit
Since I chose the 'localhost' option I enter 'http://localhost:8100' in my chrome browser window. The browser displays the Moodle Mobile app! I configure it to my local running copy of Moodle, and I'm in business!

Using the simulated mobile browser, I navigate to my test course:


I can see a "questionnaire" instance, so I select that:


And, I see that I really do need to add mobile support to the plugin!

Next posts will be me trying to make this happen.

Friday, February 3, 2017

Adding search to your Moodle plugin - post mortem

Post Mortem - Production ready search

I ended up in a lot of discussions with other Moodle developers on the best way to implement the question content search. Originally, I created a separate search area, but was hampered by the fact that I couldn't reliably keep a record per question linked to a questionnaire instance.

The search API didn't allow me to create my own index other than an integer. If it would have allowed me to construct my own, I could have stored information that identified the questionnaire and question. This resulted in the creation of the Moodle tracker item MDL-57857.

However, as we discussed this particular problem more, it really seemed that if I was searching for questionnaire instances that contained the searched for question content, then the index should really be part of the activity search. This resulted in the update I posted at the end of the last post. While that worked, the return results displayed weren't very attractive and, as was pointed out, would also include question content that may never have been displayed to the use. Questionnaires can choose questions to display based on answers to other questions.

At that point, I realized that the most likely use case for searching for question content would be for users who can access all of the question content of a questionnaire. This would be users who could actually edit the questions and users who could see all of the responses to any question.

So, it made more sense to again separate out the question search area into its own search area. This allows for two important things:

  1. Allows the administrator of the site to determine if they want to even allow question content searching.
  2. Allows the access for this content to be limited to users who have the ability to see all of the question content for a questionnaire.
You can see the final code I decided on here. The question area still stores one record per questionnaire activity, like the activity area, but it uses a different set of access rules. And, since it is a separate area, it can be enabled/disabled globally, and filtered on separately.

I believe that until there is a way to store information in the search index that allows me to go directly to the question within a specific questionnaire, this is the best solution.

Wednesday, February 1, 2017

Adding search to your plugin - Part three

Part Three - Indexing More of your Plugin

In the last post, I added more searchable content to my plugin's search indexing function, so that additional information fields specific to each instance of a plugin were included. Now I'm going to try and add question content to the search indexing.

The documentation tells me that if I'm going to index other activity data, I should use the base_mod class instead of the base_activity class. The base_mod class is located in the core file /search/classes/base_mod.php. Looking at that class, I see that it only contains one method: get_cm. All the rest come from the class it extends, base. The functions I used before, get_recordset_by_timestamp() and get_document() are abstract functions of base. This means I must provide their implementations. I also note that three other abstract functions, check_access(), get_doc_url() and get_context_url() that I will likewise need to provide implementations for. In my previous work, these three functions were provided to me by the base_activity class.

Back to the documentation, the first step it points out is that if I am going to add a new search area for my plugin, I need to provide a name for it in my language strings. Since I am going to index the questions, I add the string:

    $string['search:question'] = 'Questionnaire - questions';

to my lang/en/questionnaire.php file.

Next, I need to provide the indexed content from my questions to the search engine, using the get_recordset_by_timestamp() function. Recall that this function is responsible to construct the necessary database query for all of the content I want available for searching, execute that query, and then return the resulting recordset. For this, I create a new file called 'classes/search/question.php' and create a new question class extending the \core_search\base_mod class.

For questionnaire questions, I will need to join two tables. Question data is contained in a table that contains a reference to a survey id. That survey id is also referenced in the main questionnaire table. Since question data is not accessed outside of a survey context, I will need to create SQL that returns each question with an associated questionnaire and survey.

The function I create looks like this (full file here):

    public function get_recordset_by_timestamp($modifiedfrom = 0) {
        global $DB;

        $sql = 'SELECT qq.id, q.id AS questionnaireid, q.timemodified, ' .
            'q.course AS courseid, q.introformat, qq.name, qq.content ' .
            'FROM {questionnaire} q ' .
            'INNER JOIN {questionnaire_question} qq ON q.sid = qq.survey_id AND ' .
            'qq.deleted = \'n\' ' .
            'WHERE q.timemodified >= ? ' .
            'ORDER BY q.timemodified ASC';

        return $DB->get_recordset_sql($sql, [$modifiedfrom]);
    }

Note that I use the question id field, 'qq.id', as the first field. Moodle requires the first field of a query return to be a unique field that is used as an index for the resulting data array. Since there will be multiple questions per questionnaire and survey record, I can't use those. I will need the questionnaire id field though, so I rename it to 'questionnaireid' in the returned data. The question name and content fields, 'qq.name' and 'qq.content', contain the data that will actually be searched. This function will provide the data that will be indexed by the search function.

Now I need to provide the get_document() function. This is the function that looks at an indexed record returned from a search query and constructs the document object that is displayed on the search results screen. A document object contains a 'title' and 'content' field. I set the question 'name' data to the document 'title' and the question 'content' data to the document 'content'. The rest of the object is pretty much boilerplate from the base_activity class code and the documentation.

The function I create looks like this (full file here):

    public function get_document($record, $options = []) {
        try {
            $cm = $this->get_cm('questionnaire', $record->questionnaireid,
                $record->courseid);
            $context = \context_module::instance($cm->id);
        } catch (\dml_missing_record_exception $ex) {
            // Notify it as we run here as admin, we should see everything.
            debugging('Error retrieving ' . $this->areaid . ' ' . $record->id .
                ' document, not all required data is available: ' .
                $ex->getMessage(), DEBUG_DEVELOPER);
            return false;
        } catch (\dml_exception $ex) {
            // Notify it as we run here as admin, we should see everything.
            debugging('Error retrieving ' . $this->areaid . ' ' . $record->id .
                ' document: ' . $ex->getMessage(), DEBUG_DEVELOPER);
            return false;
        }

        // Prepare associative array with data from DB.
        $doc = \core_search\document_factory::instance($record->id, $this->componentname,
            $this->areaname);
        $doc->set('title', content_to_text($record->name, false));
        $doc->set('content', content_to_text($record->content, $record->introformat));
        $doc->set('contextid', $context->id);
        $doc->set('courseid', $record->courseid);
        $doc->set('owneruserid', \core_search\manager::NO_OWNER_ID);
        $doc->set('modified', $record->timemodified);

        return $doc;
    }

Prior to constructing the returned document object, the search API needs to verify that the requesting user has the right permissions to access the Moodle item. This is done with check_access() function. Generally speaking, what this function should do is to extract the Moodle item in the correct Moodle context, and then check that item and context with the requesting user's capabilities. In my case, a question's data is tied to the questionnaire it belongs to. So determining if a user can see that is based on their access to the specific questionnaire instance.

For questionnaire, this is complicated by the fact that a survey (which contains the questions) can belong to more than one questionnaire (in the case of public questionnaires). While this is an uncommon use, it can still exist. The check_access() function only provides me with the unique id of the data that was saved. For questions, this is the question id field. This means that I do not know which questionnaire instance I need to check when I receive this id data. So my function will need to "guess" at which questionnaire to return by looking for one that contains the question data that is accessible to the user.

The function I create looks like this (full file here):

    public function check_access($id) {
        global $DB;

        try {
            // Questions are in surveys and surveys can be used in multiple questionnaires.
            $question = $DB->get_record('questionnaire_question', ['id' => $id],
                '*', MUST_EXIST);
            $questionnaires = $DB->get_records('questionnaire',
                ['sid' => $question->survey_id]);
            if (empty($questionnaires)) {
                return \core_search\manager::ACCESS_DELETED;
            }
            $cmsinfo = [];
            foreach ($questionnaires as $questionnaire) {
                $cmsinfo[] = $this->get_cm('questionnaire', $questionnaire->id,
                    $questionnaire->course);
            }
        } catch (\dml_missing_record_exception $ex) {
            return \core_search\manager::ACCESS_DELETED;
        } catch (\dml_exception $ex) {
            return \core_search\manager::ACCESS_DENIED;
        }

        // Recheck uservisible although it should have already been checked in core_search.
        // Use the first one that is visible. Otherwise exit.
        $cmvisible = false;
        foreach ($cmsinfo as $cminfo) {
            if ($cminfo->uservisible !== false) {
                $cmvisible = $cminfo;
                break;
            }
        }
        if ($cmvisible === false) {
            return \core_search\manager::ACCESS_DENIED;
        }

        $context = \context_module::instance($cminfo->id);

        if (!has_capability('mod/questionnaire:view', $context)) {
            return \core_search\manager::ACCESS_DENIED;
        }

        return \core_search\manager::ACCESS_GRANTED;
    }

Without going into too much detail, the basic function is:
  • find the question data and all questionnaires that might contain it,
  • get a valid course module for each questionnaire,
  • find the first course module that is visible to the user and assume this is the one,
  • check that the user has the proper capabilities to view that questionnaire.
For this post, the functionality will suffice to show how this function works. Unless the function finds reason not to grant access, then granted access is returned.

(Note - I may be able to modify my functions to return a search id field that I construct to identify both the question and the questionnaire. I will look into this later, as I am not happy with "guessing" which questionnaire is used.)

The last two functions I need to provide are get_doc_url() and get_context_url(). These functions provide the link to the resulting page that will display the found content in its proper Moodle context. Since there is no way to look at only a question instance, my functions will provide a link to the questionnaire. Both functions will provide the same link. I write the code for the get_context_url() and just call that from the get_doc_url() function.

The get_context_url code looks like this (full file here):

    public function get_context_url(\core_search\document $doc) {
        $context = \context::instance_by_id($doc->get('contextid'));
        return new \moodle_url('/mod/questionnaire/view.php',
            ['id' => $context->instanceid]);
    }

Pretty straightforward.

Since I have everything I need, I load the new code, reindex all of the site contents and give it a test.

One of the first things I notice is that there is a new "search area" in the dropdown called "Questionnaire - questions":


Next, I search for a term that I know is only in some questions. Sure enough, I see the question name and the question content where matches occur. And clicking the link takes me to the appropriate questionnaire.

Updating your plugin to use the new global search feature is a fairly easy process, especially if all you do is provide the basic search. There should be no reason not to do it!

This concludes my series on adding global search to your Moodle plugin. Feel free to leave comments and suggestions. I will look at improving the question results as I noted above, and will provide updates when I have completed that.

Checklist for building additional plugin content searching:
  1. Create a new class with the name of your area in a similarly named file in your 'classes/search' directory.
  2. Add a language string name using "search:AREANAME" for the search area in your plugin language file.
  3. Create a get_recordset_by_timestamp() function to retrieve the records you want to index.
  4. Create a get_document() function to return a completed document object to the search function.
  5. Create a check_access() function to determine a matched document's availability to the user performing the search.
  6. Create a get_context_url() function to return an appropriate Moodle URL to the matched document.
  7. Create a get_doc_url() function to return an appropriate Moodle URL to the matched document.

Update - 2017.02.02
After a lot of discussion with other developers, I redid the activity search to include the question content. Since the question content was helping to find the questionnaire instance that contained it, it really made sense for it to be a part of the activity search and instance return. Since there could be many question records per questionnaire instance, I had to do the database search in the get_document() function and add the question data there. You can see the resulting work here.


Thursday, January 26, 2017

Adding search to your Moodle plugin - part two

Part Two - Indexing Custom Information

In the last post, I created the basic code required to allow Moodle's global search feature search my plugin's "name" and "intro" information fields. Now I want to extend that functionality to include the "subtitle" and "additional info" fields.

Reading the documentation, this looks like it should be an extension of the functionality I wrote for the basic case. The information fields I want to return, both belong to an activity instance. And the case I already created is for that activity instance. So I want to look at extending the class I already created.

If I decide to provide index data from other sources, then I would be creating new classes as described in the documentation following sections.

One of the concerns I have with my approach of continuing extending the base_activity class, rather than the base_mod class, is that the documentation says that I should use base_activity "to index Moodle activities basic data like the activity name and description". And that base_mod should be used for "other specific activity data". The two extra fields I want to include, aren't part of the main activity table of my plugin, but rather are stored in a second table called "questionnaire_survey". There is a one-to-one relationship though, and the information is considered part of the activity information, so I still think this the correct approach. But, to get that extra data, I will have to override the database function that gets all of that data.

Looking at the /search/classes/base_activity.php file, I see that the main function to retrieve and return the activity data is the get_recordset_by_timestamp function:

    public function get_recordset_by_timestamp($modifiedfrom = 0) {
        global $DB;
        return $DB->get_recordset_select($this->get_module_name(),
                   static::MODIFIED_FIELD_NAME . ' >= ?',
                   array($modifiedfrom),
                   static::MODIFIED_FIELD_NAME . ' ASC');
    }

This is the one I will override in my class. In its current form, it returns all of the fields from the main activity table according to the "timemodified" field. Since I want to JOIN the "questionnaire_survey" with this table, I am going to have to be selective about the fields that are returned, as there is overlap.

I open my questionnaire/classes/search/activity.php file and add the following function (full file here):

    public function get_recordset_by_timestamp($modifiedfrom = 0) {
        global $DB;

        $sql = 'SELECT q.*, s.subtitle, s.info ' .
            'FROM {questionnaire} q ' .
            'INNER JOIN {questionnaire_survey} s ON q.sid = s.id ' .
            'WHERE q.timemodified >= ? ' .
            'ORDER BY q.timemodified ASC';

        return $DB->get_recordset_sql($sql, [$modifiedfrom]);
    }

Since I haven't told the search engine anywhere that I want to include the two new fields I've added, there must still be some changes to make. I install the new code, reindex the global search and test that things work. While there are no failures, I also cannot get any matches from the "subtitle" or "additional info" fields, confirming that there is more to do.

Looking at the documentation again, the next function that is presented is get_document. From the description, this function takes a record from the query results of get_recordset_by_timestamp function, and constructs a \core_search\document object with all of the data needed for indexing. My guess, is that I need to add my two new fields here. I'm going to override this function in my class.

The main function returns a constructed object with the data added to it as variables. I believe I can override this and take the returned function from the parent, add my new data to it, and return it. When I'm done, the function looks like this:

    public function get_document($record, $options = []) {
        // Get the default implementation.
        $doc = parent::get_document($record, $options);

        // Add the subtitle and additional info fields.
        $doc->set('subtitle', content_to_text($record->subtitle, false));
        $doc->set('additionalinfo', content_to_text($record->info, $record->introformat));

        return $doc;
    }

I add the code to my site, purge all the caches and reindex the search engine. But, searching for these fields is not returning my data. On the search areas admin page, I have been using "Update indexed contents". But, since none of my questionnaire activities have actually changed, nothing is being re-indexed. Instead, I need to use the "Reindex all site contents" button. When I do this, I get an error indicating:

    Coding error detected, it must be fixed by a programmer: "subtitle" field does not exist.

It would appear that using "subtitle" is not correct.

The documentation shows an example of using get_document(), and it uses "description1" and "description2" as the fields being set. Its likely that I can't use "subtitle" and "additionalinfo" as  field names in the $doc->set() function.

The \core_search\document class is defined in the /search/classes/document.php file. The set() function uses a static array to look for specific field names it is allowed to set. The names I used, "subtitle" and "additionalinfo" are not part of those. However "description1" and "description2" are. This confirms that I need to change those functions. The set() function throws an exception when an unknown field is used, and that explains the error I saw.

My new function now looks like this (full file here):

    public function get_document($record, $options = []) {
        // Get the default implementation.
        $doc = parent::get_document($record, $options);

        // Add the subtitle and additional info fields.
        $doc->set('description1', content_to_text($record->subtitle, false));
        $doc->set('description2', content_to_text($record->info, $record->introformat));

        return $doc;
    }

When I load it, and reindex all of the site contents, my search works. I now see content found from a subtitle.


So it seems that I can add indexed fields to the a activity level search, but only two. And they need to be referred to as "description1" and "description2". I'm guessing to add more than that, I would need to create more classes using base_mod instead of base_activity.

I'll explore that more in the next post, as well as how to search for more parts of my plugin.