Add custom filters for sites

Folder /extension/lib/recorder/filters contains custom filter files that extend the capabilities of the recorder for specific sites

Simple filter url by string or regular expression

Common file for this type of filters validators.js

KellyPageWatchdog.validators[] - contains filters that do not require additional logic and associate image url links with categories by matching url with string or regular expression.

 

Example:

KellyPageWatchdog.validators.push({ url: 'deviantart', host: 'deviantart.com', /* array or string, currently used only for recomendations, may be used for filter urls by host in future */ templates: [['images-wixmp', 'imageAny']], patternsDoc: [['original-doc', 'imageOriginal']] });

Match all urls that contains url substing "deviantart".
Urls that contain substring "images-wixmp" will be marked by "imageAny" category.
For "load realed doc" procedure, all images in related document that contains substring "original-doc" will be marked by "imageOriginal" category

 

KellyPageWatchdog.bannedUrls[] - contains url strings for exclude specific urls from parsing process

Presetted categories used in associations

imageOriginal - original image

imagePreview - preview

imageAny - useful media picture (possibly original or preview)

imageByDocument - original image (same as imageOriginal, but used to identify original images in "Load related links" process)

Custom filter class with callbacks and manifest

All filter classes stored in folder /extension/lib/recorder/filters in separated files.

 

KellyRecorderFilterExample.manifest - contain several keys used to check filter compatibility using the "Load related links" feature, possibly excluding from the page by host parameter in the future.

Available keys :

• host - array of strings or a string - a list of hosts that are relevant for the filter
• detectionLvl - array with list of supported filtering options - 'imageAny' - common group for preview and originals, 'imagePreview' - filter can detect preview images, 'imageOriginal' - detect original images, 'imageByDocument' - can detect original images in related documents

 

Example :

KellyRecorderFilterExample.manifest = { host : 'example.com', detectionLvl : ['imageAny', 'imagePreview', 'imageOriginal', 'imageByDocument'] }; /* enable filter and adds filter to common filters list */ KellyPageWatchdog.filters.push(KellyRecorderFilterExample);

Manifest declare that filter works on sites with the host example.com and recognizes original image links (auto adds category imageOriginal), previews image links (imagePreview) and is able to find originals from a preview (imageByDocument)

addItemByDriver

KellyRecorderFilterExample.addItemByDriver( handler?: KellyPageWatchdog, data?: object {el?: DOM element, item?: object {}}, )

Method is called before data.el DOM element default parsing by kellyPageWatchDog.parseItem

Arguments:

• handler - instance of the kellyPageWatchDog class
• data - object with structure {el, item}, where el is the DOM element processed on the page (every Node.ELEMENT_NODE document element is checked, the exception is 'script', 'iframe', 'frame', 'include-fragment', 'svg' ), item - the current processing object with the images collected in it, created based on the DOM of the el element

 

Return handler.addDriverAction

 

You can search image links by kellyPageWatchDog methods

handler.addSrcFromAttributes(el, item, excludeAttributes)

handler.addSrcFromStyle(el, item, groups)

handler.addSingleSrc(item, src, contextStr, el, groups)

 

In build in methods quality of image links (item.relatedSrc) is determined based on several conditions

• Level of trust for a specific tag (IMG | SOURCE | DIV ...) depending on the tag name and attribute key
• Element attribute name
• Ability to check the file extension based on the link string

 

Example:

KellyRecorderFilterExample.addItemByDriver = function (handler, data) { /* check if its needed site url, check data.el */ if ( handler.url.indexOf('example.com') != -1 && (data.el.classList.contains('photo') || data.el.classList.contains('thumb')) ) { /* add single image to item.relatedSrc, from data.el */ handler.addSingleSrc(data.item, data.el.src, 'addSrcFromAttributes-src', data.el, 'imagePreview'); /* if no images was added prevent default behavior by return handler.addDriverAction.SKIP and add data.item to common handler.imagesPool array on success */ return data.item.relatedSrc.length > 0 ? handler.addDriverAction.ADD : handler.addDriverAction.SKIP; } }

parseImagesDocByDriver

KellyRecorderFilterExample.parseImagesDocByDriver( handler?: KellyPageWatchdog, data?: object {thread}, )

Method is called on related link document (item.relatedDoc) succesful loaded

Arguments:

• handler - instance of the kellyPageWatchDog class
• data - object {thread}, where thread - result object that contain request controller with result data

 

Example:

KellyRecorderFilterExample.parseImagesDocByDriver = function(handler, data) { /* is matched url, and response is object (response "content-type" = json) */ if ( handler.url.indexOf('example.ru') != -1 && typeof data.thread.response == 'object' && handler.url.indexOf('original-image-page') != -1 ) { if (data.thread.response.originalImage) { /* add image directly to images array (skip handler.addSingleSrc and such) */ handler.imagesPool.push({relatedSrc : [data.thread.response.originalImage]}); /* prevent default related document parser */ return true; } /* is matched url, and response is setted */ } else if (handler.url.indexOf('example.ru') != -1 && data.thread.response) { var parser = new DOMParser(); var doc = parser.parseFromString(data.thread.response, 'text/html'); var src = doc.querySelector('[property="og:image"]').getAttribute('content'); if (src) handler.imagesPool.push({relatedSrc : [src]}); } }

onInitDocLoader

KellyRecorderFilterExample.onInitDocLoader( handler?: KellyPageWatchdog, data?: object {docLoader?: KellyLoadDocControll, hostList?: []}, )

Event called before load related documents process. Some of configuration params for docLoader can be updated. If return "false" - stops load related document process

Arguments:

• handler - instance of the kellyPageWatchDog class. Default instance used - location (url, host, hostname) setted from first recorded tab
• data - object {docLoader, hostList}, where docLoader - related documents requests controller, hostList - array of all hosts for current image list

 

Example:

KellyRecorderFilterExample.onInitDocLoader = function(handler, data) { data.docLoader.parser.updateCfg({ pauseEvery : '50', pauseTimer : '1.2,1.8,2,2.4,2.8', maxThreads : '1', }); }

onInitLocation

KellyRecorderFilterExample.onInitLocation( handler?: KellyPageWatchdog, data?: object {url?: string, host?: string, hostname?: string}, )

Event calls after init parser controller (click on "Record" / "Images from Tab")

Arguments:

• handler - instance of the kellyPageWatchDog class
• data - object {url, host, hostname}, url - current parsed document (tab or related document link) url, host - url.hostname + protocol, hostname - url.hostname

 

Example:

KellyRecorderFilterExample.onInitLocation = function(handler, data) { /* handler.url, handler.host see all accessible variables in kellyPageWatchDog */ }

onInitOptions

KellyRecorderFilterExample.onInitOptions( handler?: KellyPageWatchdog, data?: object {options?: KellyOptions}, )

Event calls on options page shown

Arguments:

• handler - instance of the kellyPageWatchDog class. Default instance used - location (url, host, hostname) setted from first recorded tab
• data - object {options}, where options - instance of KellyOptions (KellyOptions.js)

validateByDriver

KellyRecorderFilterExample.validateByDriver( handler?: KellyPageWatchdog, data?: object {item?: object {}}, )

Calls after all parsing process by parseItem, addItemByDriver when data.item is ready

Arguments:

• handler - instance of the kellyPageWatchDog class
• data - object {item}

 

Example:

KellyRecorderFilterExample.validateByDriver = function(handler, data) { /* any changes for data.item {relatedSrc, relatedGroups, ...} return false; - prevent adding data.item to common pool */ }

onStartRecord

KellyRecorderFilterExample.onStartRecord( handler?: KellyPageWatchdog, data?: object {context?: string}, )

Calls before parsing process

Arguments:

• handler - instance of the kellyPageWatchDog class
• data - object {context}, where context - string that represent initiator action ("parseImages" - click on "Images from Tab", "isRecorded" - continue after page \ tab loads, "startRecord" - click on "Record")

 

Example:

KellyRecorderFilterExample.onStartRecord = function(handler, context) { if (handler.url.indexOf('example.ru') == -1) return; handler.additionCats = { example_comment : {name : 'Comment', color : '#b7dd99'}, example_post : {name : 'Post', color : '#b7dd99'}, }; }